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ABSTRACT 


The exact sampling distribution of reliability estimates of 
a composite test is known only for the case when the test scores of 
the parts can be expressed in a linear model and satisfy all the assump- 
tions of the two way mixed model ANOVA with one observation per cell 
including normality of true and error scores. For more general cases, 
the sampling distributions have been in general unknown or ignored by 
the psychometricians. 

This study examined the more liberal concepts of test theory 
and reliability in terms of the underlying models and assumptions, and 
investigated the sampling distribution of reliability estimates by 
performing a number of computer simulated sampling experiments under 
various models and distributional assumptions for true and error scores. 
The models employed were a mixed model ANOVA, essentially t equiva- 
lent measurements, congeneric and multi-factor true score models for 
continuous cases, and the normal ogive model for binary item cases. 
For the distribution of true or latent and error scores, uniform, 
normal and exponential distributions were used. 

The most general model was found to be a multi-factor true 
score model and all others could be shown to be special cases of this 
model. The most important factors influencing the sampling distribu- 
tion are found to be uni-factorness and normality of true scores for 
continuous cases, and homogeneity of item difficulty parameters for 
binary cases. The distributions of error scores were found to be 


unimportant for both cases. 
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To determine the robustness conditions of the traditional 
F-test, the empirical distributions obtained by the sampling experi- 
ments were compared with those theoretical distributions obtained under 
the ANOVA and normal theory model. A number of conditions for robust- 
ness are given. 

A new formula for the standard error of reliability estimates 
is introduced by analytical means and the validity of the formula was 
examined through computer simulated experiments. The new formula was 
found to be superior to traditional formulas based on normal theory 
when the normality of true score is not valid. Though the formula is 
derived under the ANOVA model, it was also found to be better than the 
traditional formulas under more general models. 

Implications of these findings to test theory ae applications 
are discussed and some numerical examples are given to show how the 
findings and the computer programs developed might be applied in 


practical situations. 
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CHAPTER ONE 


INTRODUCTION 


1.1.0 The General Problem 


The estimation and interpretation of reliability has been a 
central issue for psychometric theorists and test authors as well as 
the users of educational and psychological tests. The reliability 
coefficient of a test, a population parameter, is defined as the ratio 
of the true score variance due to individual differences of the subjects, 
to the total test score variance in a population for which the test is 
developed. 

A number of formulas for measuring reliability have been 
derived by many theorists since the initial formulation by Spearman 
(1910) of his theory of true and error scores. Most of the formulas 
express reliability as a function of the moments. of the part scores 
and the total test scores under assumptions of parallel or equivalent 
measurements among the part tests, or, as a correlation coefficient 
between the observed test scores and a second set of scores on a real 
or hypothetical variable. In most cases, the formulas involve only 
point estimation of the reliability, and have been obtained by sub- 
stituting the sample moments of the part-scores and the total scores 
into formulas which are valid in the population. Statistical 
properties of such estimates are in general unknown or ignored. 

Investigation of the distribution of reliability estimates 
requires a mathematical model and a number of assumptions. The 


validity of the estimates of reliability largely depends on the validity 
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of the model and underlying assumptions. Even for rather rigorous 
models and assumptions the sampling distributions are unknown except 
in some special cases. 

For a valid statistical inference about a population parameter 
the sampling distribution of the parameter estimate must be known, and 
the reliability cannot be an exception. For example, if a standardized 
test has been administered to a sample of subjects, it is sometimes 
necessary to compare the sample reliability estimate with the reliability 
claimed by the test authors, i.e., it is desirable to know whether the 
difference between the two values can be attributed to sampling fluctu- 
ations, or, whether there is a significant difference due to population 
difference. If a test is administered to two independent samples of 
subjects, a comparison of the two estimates of reliability may be 
necessary to determine the underlying cause of any observed difference. 

The standard error of the reliability estimate is another 
useful measure of the precision of the estimates, but without any 
knowledge of the sampling distribution of the estimates, confidence 
intervals for the population reliability are impossible to calculate. 

Most of the available formulas for reliabtlity estimate 
depend on the estimation of variance components, using various, 
explicit or implicit, parallel or equivalent test form assumptions 
among the part test scores. Even though the estimates of the variance 
components thus obtained are usually unbiased, the estimates of the 
reliability are, in most cases, biased, and the statistical properties 
are unknown. 


Since the early years of test theory, it has been 
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recognized by theorists and test users that the calculated reliability 
is, in fact, nothing more than an estimate of the true or population 
reliability, and therefore subject to sampling fluctuations. Even 
with this recognition, little work has been done to investigate the 
distribution of such estimates. 

This study will investigate properties of the sampling 
distribution of reliability estimates based on Alpha or KR20 formulas 
using computer simulation techniques, and will employ various models 
and distributional assumptions for true or latent, and error scores 


described in the following two chapters. 


1.2.0 Review of Related Studies 


[os ae | Concepts of Reliability 


Even during the initial developments of test theory, psychol- 
ogists showed interest in the formula for the standard error of reli- 
ability estimates as an indicator of the precision of such estimates. 
During this period most psychologists interpreted reliability as 
a correlation coefficient between classically defined parallel measures. 
Using this definition, attempts were made to apply the well known 
sampling distribution of correlation coefficient with the assumption 
of a bivariate normal distribution. However, unlike the usual inference 
about correlation coefficients, in most cases, the population relia- 
bility is considered to be close to unity rather than zero, and hence 
its distribution has extreme negative skewness making the usual 
normal approximation of little use (Jackson and Ferguson, 1941, p. 2) 


When the split half method was introduced, the reliability 
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estimate was seen to depend on the way the test was split. As a result, 
the reliability estimate based on Alpha or KR20 was considered to be 
superior to the split half estimate since the former gave a unique 
estimate. 

Cronbach (1951) has shown that Alpha or KR20 is an average 
of all possible split half reliabilities in the population. He 
thoroughly investigated the coefficient Alpha from the point of view 
of factorial structure. The Alpha coefficient was interpreted as the 
proportion of the test variance due to all common factors among the 
part scores, and as an index of consistency, an estimate of first 
factor concentration. He also showed that Alpha is a lower bound of 
the test reliability, but did not explicitly discuss the sampling 
aspect of the Alpha estimate. 

The concept of test reliability has been under continuous 
change: the classical concept based on parallel tests has been 
modified and the assumptions relaxed. Burt (1955) and Tryon (1957) 
initiated a new concept of domain sampling, and the reliability as 
an index of generalizability has been advocated by Rajaratnam (1960), 
Cronbach, Rajaratnam and Gleser (1963), and Rajaratnam, Cronbach and 
Gleser (1965). Their conceptual framework relied heavily on ANOVA 
models, and initiated a process of liberalization of reliability 
theory from the rather restrictive classical orthodoxy of test 
parallelism. However their efforts concentrated on the problem of 
point estimation, and little attention was paid to sampling aspect 
of the estimates. 

Lord and Novick (1968, p. 50) defined the concept of 


‘essentially +t equivalent measurements', and Novick and Lewis (1967) 
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have shown that the coefficient Alpha is identical to the reliability 
coefficient if andonly if a test consists of essentially +t equiva- 
lent parts. If this condition is not satisfied Alpha is a lower bound 
for the reliability, confirming previous studies of Guttman (1945, 1953), 
Cronbach (1951), and others. To evoke the principle of essentially +t 
equivalent parts, it has been argued that only true score variances need 
be identical, i.e., homogeneity of true score variances, and not identical 
error score variances nor identical true scores among the part tests. 

The assumptions of classical parallel tests are, therefore, relaxed 
substantially. Jdéreskog (1968, 1970, 1971) defines the concept of 
congeneric test scores which measure the same trait except for errors, 
relaxing the essentially +t equivalent measurement conditions further. 
Under this model any pair of such tests have linearly related true 
scores. The sampling distributions of the reliability estimates under 
these models are not yet generally known. 

As an alternative to conventional uni-factor true score 
models, a multi-factor true score model has been advocated by LaForge 
(1965) using the multiple factor analysis model. An estimate of the 
squared multiple correlation of a part score with the scores of 
remaining parts, which is one estimate of the test communality in 
factor analysis, is proposed as an estimate of the reliability of the 
part score. 

For certain kinds of tests, especially in the field of 
achievement tests, this approach seems more reasonable than the 
conventional uni-factor approach, but the old controversial problem 
of determining the number of factors in a factor analysis must still 


be resolved. However, the multi-factor model provides a general model 
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for computer simulation purpose, as will be seen in the next chapter, 
since the ANOVA and other models may be considered as special cases 


of the multi-factor model. 


hae 22 Sampling Theories of Reliability Estimates 


Lord (1955) defined three kinds of sampling arising in test 
theory; sampling of subjects (Type 1), part tests or items (Type 2), 
and a simultaneous combination of the two (Type 12). Lord also 
discussed the sampling distribution of KR20 under Type 2 sampling 
without the presentation of the standard error of the KR20 estimates 
in terms of the population parameters. 

A statistical sampling theory of the reliability estimates 
has been made possible through the application of ANOVA techniques to 
test theory. Hoyt (1941), Jackson and Ferguson (1941), Ebel (1951), 
Burt (1955), Cronbach, Rajaratnam and Gleser (1963), Feldt (1965, 1969), 
Maguire and Hazlett (1969) and many others investigated the reliability 
estimate under some form of ANOVA models. However, most of their 
discussion was limited to point estimation and little attention has 
been paid to the sampling fluctuation of the estimates or interval 
estimates. 

Since the ANOVA models usually provide unbiased, consistent 
estimates of the variance components by some linear combination of 
various mean squares, the reliability estimates thus obtained are in 
general consistent estimators. But, in most cases, they are biased 
and do not have the desirable minimum variance property. 

Although Jackson and Ferguson (1941, p. 40) related the 


F-statistic to the so called ‘sensitivity of a test', or the square 
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root of the commonly referred signal-noise ratio, it was Ebel (1951) 
who first explicitly linked the sampling distribution of the reli-- 
ability estimate itself to an F-statistic. He applied the concept of 
'intra-class' correlation coefficient to a rating data set, and by 
employing the well known F-distribution, has shown a way to obtain 
confidence intervals of the population intra-class correlation which 
he interpreted as the reliability of a judge. However, the assumptions 
underlying the ANOVA model were not explicitly specified. 

Kristof (1963) presented a rather complete sampling theory 
of reliability estimates within the context of the assumptions of 
classical reliability theory with the exception that the means of the 
part tests were allowed to be different, i.e., the part test scores 
are 'essentially' parallel measurements. Under Type 1 sampling and 
the assumption of a multi-normal distribution of the part test scores, 
a maximum likelihood estimator of the common correlation among the 
parts was obtained, i.e., the intra-class correlation coefficient. It 
was shown to be biased. A bias-free formula was introduced and the 
sampling distribution of the estimates based on this formula is shown 
to be related to the F-statistic. A method of statistical inference 
about the intra-class correlation, which was interpreted as the 
reliability of a part test, was suggested. Kristof's results are in 
close agreement with those obtainable under an ANOVA model. He has 
also showed that the estimate of the Alpha coefficient, in terms of 
second moment sample statistics, is the same as the maximum likelihood 
estimator of the reliability when a test has been divided into 
essentially parallel parts, with an assumption that the parts have a 


multi-normal distribution. 
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Kristof (1964) also investigated the distribution of 
reliability estimates for the first time without relying on the 
classical equal variance assumptions among the part test scores. A 
working formula for testing the significance of the difference between 
the two reliability estimates was derived under the assumption that 
each part has been administered to the same sample of subjects and 
that each part test could be split into parallel halves. He also 
investigated (1969, 1970) the sampling distribution of reliability 
estimates under the multi-normal assumptions when a test has been 
split into two parts not necessary parallel in the classical sense. 

A likelihood ratio test of the point hypothesis concerning the popu- 
lation value of Alpha was derived. This method was then used to yield 
confidence intervals for the parameter for any chosen level of 
confidence. 

For the case where the parts of a test are simply binary 
items, the sampling distribution of KR20 estimates is more complicated 
than the case of continuous part scores. Most theorists have assumed 
an intermediate hypothetical variable between the item response and 
underlying true or latent trait score and linked the two variables with 
the help of the intermediate variable and a mathematical model. Lord 
(1952), and Lord and Novick (1968) used a normal ogive model, while 
Birnbaum (1967, 1968) proposed logistic, Poisson and other mathematical 
models. Although the latent trait approach provides means for investi- 
gation of the relationships among the item parameters and the test 
scores, nothing analytical has yet been done for the distributional 
theory of reliability estimates or its application even with the 


restrictive mathematical models and assumptions. 
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Aoyama (1957) has given explicit formulas, in terms of 
population parameters without any distributional assumptions, for the 
expected value and variance of KR20 estimates for Type | and Type 2 
sampling situations. These results clearly indicate that the estimates 
are biased. However the formulas involve some approximations and cal- 
culation of higher order moments, and are too complex for any practical 
use. 

Since the exact sampling distribution of KR20 estimates is 
not obtainable by analytical means, some researchers have attempted to 
approximate it by an ANOVA model. Feldt (1965) has investigated the 
applicability of the ANOVA model. He pointed out that imposition of 
a one-zero scoring scheme violates such assumptions of the ANOVA model 
as continuity of the scores, homogeneity of error variances and 
independence of true and error scores. He compared the results obtained 
under the ANOVA model with an empirical distribution based on real data 
reported by Baker (1962), and claimed the model robust when the 
assumptions are violated. Feldt referred to the model as a two way 
random effects model, but actually it was a mixed model as will be 
seen in the following chapter. Further applications were made of the 
method by deriving a scheme for testing the equality of two KR20 
coefficients based on two independent samples using an approximate 
distribution of the product of two independent F-statistics (Feldt, 
1969). Cleary and Linn (1968) adopted the same method as Feldt and 
gave an explicit formula for the standard error of KR20 estimates. 
However, their results are heavily dependent on normality assumptions 
which are not satisfied for KR20 cases. 


Except for the case of approximation by employing unrealistic 
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assumptions of the ANOVA model for essentially parallel tests, most of 
the attempts to obtain the sampling distribution of KR20 estimates 
have either failed or resulted in unuseable formulas such as given by 


Aoyama (1957). 


eZ. 3 Empirical Approaches 


The use of empirical approach to solve a statistical problem 
is as old as statistics itself. For example ''Student'' (1908) derived 
the analytic expression for the t-statistic and also established the 
validity of his argument by a sampling experiment. In education and 
psychology, a number of empirical investigations have been performed, 
with or without the help of a computer, to ascertain the robustness of 
the F-test when certain assumptions underlying an ANOVA model are not 
satisfied. Norton (1950), Boneau (1960), Hsu and Feldt (1969), and Bay 
(1970) are some examples of such investigations. 

In reliability theory, Baker (1962) investigated a sampling 
distribution of KR20 estimates under Type 1 sampling constraints by 
actually performing experiments using real test results. Payne and 
Anderson (1968) tabulated the sampling distribution of KR20 estimates 
based on computer simulation. However, their experiments were limited 
to the cases of equal item difficulty parameters and inter-item 
correlations, i.e., phi coefficients. 

Nitko and Feldt (1969) performed a computer simulation 
study of KR20 estimates and reported that, in contrast to general 
belief, the effect of item difficulty is minimal. Nitko (1968) 
employed the same method to investigate power functions for the test 


of significance of KR20 in one and two sample cases as proposed by 
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Feldt (1969). Weitzman (1967) reported the result of a simulation 
of test-retest reliability of a multi-choice test assuming a beta 
distribution of true scores. Shoemaker (1966) also used a computer 
simulation model to investigate the estimate of Cronbach's gener- 
alizability coefficient for unmatched data to clarify the extent to 
which stratification must be taken into account in the choice of the 


generalizability formula. 


fee 4 Summary 


Recently, the concept of reliability has been modified and 
the restrictive classical assumptions of parallel tests relaxed 
substantially. However, the sampling distribution of reliability 
estimates based on Alpha or KR20 formulas are in general unknown 
except for the case when the unrealistic ANOVA model and underlying 
assumptions are used. 

A number of fragmental attempts have been made recently to 
investigate the distribution by empirical methods, but there is no 
overall study into the statistical properties of the distribution under 
the more liberal concept of reliability either by analytical or 
empirical means. 

The purpose of the present study is to investigate the 
statistical properties of the sampling distribution of reliability 
estimates when the classical parallel tests or more recent ANOVA 
models and the assumptions underlying these models are not all 
satisfied. More liberal concepts of reliability are to be examined 
in terms of models and assumptions underlying them, and sampling 


distributions under these models with various distributional assumptions 
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not necessary normal will be investigated by employing computer 
simulated statistical experiments. Comparisons are to be made of 

these results with those obtainable theoretically from the ANOVA model 
and normal theory to see the robustness of the theoretical distributions 


against the violation of assumptions. 


1.3.0 Some Preliminary Specifications and Notations 


3.1 Specifications 


(a) With some exceptions, Greek letters will be used to 
denote population values, while the observations and sample quantities 
are denoted by Roman letters. To make notation simpler, no attempts 
are made to distinguish random variables from their observed values. 

(b) Scalars will be denoted by capital and lower case 
letters, matrices will be denoted by underlined capital letters, and 
column vectors by underlined lower case letters. Row vectors will be 
indicated by transpose of column vectors, i.e., by priming them. 

(c) An estimator of the population parameter and its value 
will be indicated by placing a caret or 'hat' over the parameter. 

(d) The normal distribution with mean wu and variance 
e will be denoted by N(p, rae In general, a J-variate normal 
distribution having a mean vector wy and a dispersion or variance- 
covariance matrix 2 will be denoted by N(u, Du The chi-square 
statistic with n degrees of freedom and the F-statistic with n 
and m degrees of freedom are denoted by x and Jae respectively. 

(e) The expectation, and dispersion operations for a 


vector random variable x will be denoted by E(x) and D(x), 


namely, 
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D(x) = Cov (x35) Cov (x3 5X5) 

Cov (x) 4x)) ‘ . var (x )) 


where E(x.), Var (x.), and Cov (x; 4x5) denote the usual expec- 
tation of Xi variance of Xi and covariance between Xs and 
x; respectively. 

(f) The correlation coefficient between two random variables 


x and y is denoted by Cor (x,y), namely, 


Cov (x,y) 


Cor (x;y) <= : 
(Narco Varny) 1 


(g) An identity matrix is denoted by |, while a vector 
of length J whose elements are all 1's is denoted by 1. 

(h) Dot subscripts are used to indicate sample means. 

(g) Braces, { }, are used to indicate all elements in a 


set of variables. 
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1.3.2 Notations 


The following is a brief glossary of important symbols used 


frequently. 


e 


kJ 


indexing subscript for subjects in the sample, i = 1,2,...,|I 
indexing subscript for subjects in population, k = 1,2,. 
indexing subscript ‘for parts (items) of a test, j = 1,2,...,d 
Sample size, a fixed constant 

number of parts (items) in the test, a fixed constant 

the observed score random variable of subject i on the jth 
part test; it stands for the corresponding response strength 
variable for the binary item test 

the observed score random variable of subject i on the jth 
item, takes on values one or zero 

the true score of subject i on the jth part 

the error score random variable of ae 

the true score of subject i after adjustment is made for 
difference in difficulty levels among the J part tests 

the effect or ability level of subject i in deviation 
form, m.-u 

the fixed effect of jth part, or the threshold constant for 
jth item; indicates the difficulty level of jth part (item) 
the expected value of m, over the population 

the variance of m, over the population, assumed to be 
common to all J parts under ETEM assumption 

the variance of a over the replications, assumed to be 


common to all subjects for all specific part j; defined in 


terms of response strength variables for the binary case 
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common value of oO among the J parts under the 
homogeneity of error variance assumption 

the unweighted sum of J part scores for subject i 

the unweighted sum of J items for subject i 

the variance of the jth part (item) score 

the variance of Yi 

the variance of x; 

the regression coefficient of a on f. under the uni- 
factor true score model, or the standard deviation of the 
true score of the jth part; the biserial correlation between 
x.. and f. for the binary item case 


VJ 


the tetrachoric correlation between items j and j' 


the inter item correlation coefficient between items j and 


standardized error random variable, i.e., 


ee oem is 
VJ SE) 


standardized true score random variable for continuous 
case i.e., a, = Ont; for the binary item case the 
latent or factor score 

the factor loading matrix of size Jxr 

the number of factors of the true score 

item characteristic function of the jth item 

item difficulty of the jth item 

the normal density function 

the cummulative normal distribution function 

the reliability coefficient of the jth part 


the reliability coefficient of the unweighted sum of 


J parts (items) 
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Yr the kurtosis of true score a, or f 
Y the kurtosis of error score e., or 


Y the kurtosis of test score Y; 


1.3.3 Vectors and Matrices 


The following vectors and matrices are used frequently. 
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jm 


on the mth 


is the factor loading or regression coefficient of vag 


true or factor score under a multi-factor model. 


1.3.4 Definitions 


The following is a short list of definitions for the most 


often used terms in this paper. 


ANOVA Model 


Parallel 


Essentially 
Parallel 


(ANOVA) 


Unless specified otherwise, this term refers to a 
two way mixed model analysis of variance linear model 
with one observation per cell. The levels of the 

row factor stand for the subjects in the sample, and 
the effects are assumed to be random, while the 
levels of the column factor stand for part tests or 
items of a composite test, and the effects are 


assumed to be fixed. 


Two measures are said to be classically or strictly 
parallel if (a) the test score may be considered 
consisting of two independent parts, true and error 
scores, (b) true scores are identical, and (c) error 
and total scores have idenical means and variances 


for each of the two measures. 


The same as parallel measures except that the true 
scores may differ by a constant. The true, error 

and test scores have identical variances, but the 
means of the true scores may differ. Under the ANOVA 


model, the measurements are essentially parallel. 
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t Equivalent 


Essentially 


t Equivalent 


(ETEM) 


Congeneric 


True Score 


Multi-factor 


(M.F.) 


The same as parallel measures except that the 
variances of the error scores may differ. The 
variances of test scores may differ, but the 


means must be equal. 


The same as t equivalent measurements except 
that the true scores may differ by a constant. 
The variances of true scores must be identical, 


but means and variances of test scores may differ. 


The same as essentially t equivalent measurements 
except that the true score is required only to 
measure a single trait. The true scores of two 
measures are linearly related, but their means 


and variances may differ. 


The same as congeneric true score case except that 
the tests measure more than one trait, i.e., the 
factorial structure of the true score could be 


more than one factor. 


The above definitions of different but related types of 


measurements are compared in Table 1.1. 
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TABLE 1.1 


Comparisons of the Definitions of Various Measures 


Type of True Scores Error Scores Test Scores 
Measures 


Score! Meant 4 Mean Var. Mean Var. 


Parallel 


Essentially 
Parallel (ANOVA) 


t Equivalent 


Essentially T 
Equivalent (ETEM) 


Congeneric 


Multi-Factor 
(M.F.) 


Note: lieve scores for the same subject. 


-Means in the population. 


3variances in the population. 


h 


1: Identical among the measures. 


5 


D: May differ among the measures. 
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CHAPTER TWO 


TEST MODELS FOR THE CONTINUOUS PART SCORE CASES 


Two distinct cases may be considered for a theory of relia- 
bility: the first is the case of continuous observed scores for the 
parts of a test, and the second is the case in which the scores of the 
parts are 'counter' or 'indicator' variables, i.e., a one is assigned 
for a correct response and zero for a wrong response. Due to the neces- 
sity of a different statistical treatment for each of the two cases, 
only the continuous case is duscussed in this chapter. The discussion 
is also limited to Type 1 sampling situations. The binary item 
situation will be discussed in the following chapter. 

For the continuous score case, ANOVA type linear models 
are the most powerful and have a wide range of applicability. From 
among many possible models, the discussion is limited to a two way 
mixed model ANOVA with one observation per cell. Generalization to 
other more complex designs is a straight forward matter, however, 
complexity and difficulty of interpretation is a problem due to inter- 


action effects. 


2.1 ANOVA Model 


A test consisting of J parts (J > 2) is considered under 
the strict parallelism assumptions among the J part tests except that 
the means of the J parts may differ by a constant due to the 
difference in the difficulty levels of the parts. If the test is 


administered to a random sample of | subjects (I ae the observed 
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score of the ith subject in the sample on the jth part, a random 
variable denoted by vey? may be written in a linear form in accordance 


with the classical theory of true and error scores, namely, 
al) WY ocr, See Fanner, gy a fee WW e2e wes 


where ae and eng denote the true score and error score respectively. 
An infinite idealized population of subjects denoted by P, 

from which the sample of | subjects is supposedly drawn, is hypo 

thesized and the findings on the sample are to be generalized to the 

population. Labelling the subjects in P as k (k= 1,2,...), the 

score Vij may be conceptualized as the realization of a random 

process which may occur under repeated measurements on a single 

subject, labelled by k, ona fixed part test j with the assumption 

that the subject does not change or 'learn' over the repeated measure- 

ments, that is the replication is under experimentally independent 


conditions. Then the true score T may be considered the mean of 


kj 
Yqj over replications, or the expected value of "he over the 
distribution of Yigj for fixed k, or over the so-called ‘propensity 
distribution' of Vij (Lord and Novick, 1968, pp. 29-30). Mathematically 

Thy may be defined as the expectation of the random variable Yj? 
for given k and j. The elements of ine have a joint distribu- 
tion with respect to k in the population P and the number of 
replications. 


By the assumption of parallelism, the true score ci may 


be written, 


(22) ee m(k) + 8 , 
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where af is a fixed constant specific to the jth part test represent- 
ing the difficulty level of the part relative to the other parts, and 
m(k) is the adjusted true score for subject k indicating the real 
ability lavellclo? the subject, and assumed to be independent of j. 
Thus, m(k) may be considered a random variable with repsect to k 
distributed over the population P. 

Since the {85} indicate only the relative difficulty 
levels among the J parts, without loss of generality it may be 


assumed that, 


t2=3) yep en gOe. 


The labels of the subjects in the sample may be given by 


{ky skos++- sk, 


score of the ith subject in the sample. If wu and on denote the 


}, and m. = m(k.) where m, is the adjusted true 


mean and variance of the adjusted true score m(k), then they are 
the expected value and variance of the random variable m(k) cal- 
culated with respect to the distribution of k in the population. 
Since each of the | subjects may also be considered as a randomly 
selected subject drawn from | identical populations with mean 4 
and variance ie one subject chosen per population, each of the 
{m, } may also be considered as a random variable distributed indepen- 
dently and identically with expected value yu and variance On: 

The variance of the error random variable Stee calculated 
with respect to the propensity distribution of Yj? for fixed k 
and j, shall be denoted by of (k). Although it is conceivable 


that the brighter subjects with higher m(k) might respond to the 
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test more consistently over replications, and have smaller variances, 
for the present discussion, it is assumed that 08; (k) is the same for 
all subjects in the population and the common error variance is denoted 
by o.., which depends only on j. This assumption is rather restric- 
tive, but it is necessary since only one set of part test scores is 
assumed to be available for each subject in the sample, and therefore 
8; (k) would not be an observable quantity without this assumption. 
Furthermore, following the assumptions of classical paral- 
lelism (e.g., Gulliksen, 1950, pp. 14-25), under the ANOVA model, it 
shall also be assumed that the error scores te, i} have expected 
value zero and equal variance, denoted by Oo, for all the J _ parts, 
2 homogeneity of error variance is also assumed among the part tests. 
In addition they are assumed to be independently and identically 
distributed, and independent of {m(k)}. 


The effect of a subject labelled k in the population is 


defined as, 
(2.4) a keer am (kes one 


such that the effect of the ith subject in the sample, denoted by 


aes 
(2.5) a. = on. 
Applying (2.2) and (2.5), (2.1) becomes the basic model equation, 


(2.6) ies aa 8 + ei , 


with the following assumptions, 
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(aj@ Ca. vand eH are independent random variables, 
(b) ) 8, = 0, 


(c) ta. } are identically distributed with E(a,) = 04 


Zz 


C2 7) and Var (a,) SRC 


(d) fe} are identically distributed with Ele; 5) =20:, 


and Var (e; ;) oe 


Thus the expected value and variance of an observation A 


(2.8) Ey.) a= et BF Var (v5) = on + o 


If yy denotes the unweighted sum of the J part scores 


for subject i, namely, 


(2.9) ee we ie ee Gaeta 
J J 
then, 
(2.10) E(y,) = Ju; Var (y,) = Ie of + J o 


The reliability of a test is defined to be the ratio of the 
variance due to individual difference or the ‘'effect' of subjects to 


the total test score variance (Lord and Novick, 1968, p. 61). Fora 


part j, 
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where 6 = o,/o, is the so-called signal-noise ratio or the square of 
sensitivity of a part test score (Jackson and Ferguson, 1941, p. 40). 
For the total score, 


(2-21,2,) ee Pe ry ees Oo 


Because, 
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= on ’ 


and the correlation coefficient between part j and j' (j # j') is 


lL, 
Cor (Yi peYige Cov (ype; j.)/[Var (yi) Var Yi T 


on/ (a4 + 0°) mec 
The common reliability among the J _ parts, Py is the so-called 
'intra-class' correlation coefficient which is the ordinary correlation 
coefficient between the part scores 4 and Yaya under the ANOVA 
assumptions above. Or alternatively, % is the square of the index 
of reliability, which is the correlation between a and ais since, 


cov (y;,.a;) = Ella, +e,,)(a)] =o , 
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Under this model no assumption of random sampling of parts 
is required. The models used by Hoyt (1941), Ebel (1951), Winer 
(1962, p. 124), Lord (1964), Feldt (1965), and Maguire and Hazlett 
(1969) are essentially the same as this, although some treat the fixed 
effect {B53 as random effects assuming the existence of a population 
of part tests and random sampling of J parts from it. As has been 
shown, the assumption is not necessary. 

By the usual mathematical presentation (e.g., Scheffé, 1959, 


p. 261) the unbiased estimator of on and 0” are given by, 
6° = (MS, « HS_)/J; o” = MS 
A A e , ; 


where MS) and MS. are mean squares for subject effects and errors 


respectively. They are obtainable from an ANOVA table given as the 


fol lowing: 


TABLE 2.1 


Two Way Mixed Model ANOVA Table 


Subject Sap sees MS ,=SS,/ (1-1) 


MS,=SS,/ (J-1) 


Parts 


Errors MS. _ SS_/v 
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substituting the unbiased estimator of variance components into (2.11) 


or (2.12), 


ar MS, - MS 


oe z A e aiae Ghikae 
(a) “y 7 22 n2 MS, + (J-T)MS Peratninen| 
oy, + om A e 


(2213) 


where F is the ratio of mean squares, namely’ F = MS/MS, : 

The derivations up to and including equation (2.13) are 
valid without any distributional assumptions on {a;} and fo 
In order to obtain a sampling distribution of the estimate (2.13), 
distributional assumptions are necessary. The simplest normal assump- 


tions are 


(a) all {a,} are distributed as N(O,0%) 
(2.14) 
(b) ala fe; j} are distributed as N(0,02) : 


With the above assumptions, model (2.6) is identical to the 
two way mixed model ANOVA with one observation per cell (Scheffé, 1959, 
2 2 Z 
p. 261). It can be shown that SS ,/ (Jo, + o.) and SS /o, are 
distributed as chi-square with I-1l and wy degrees of freedom respec- 


tively, or 
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= PL RMANE NE: Vids be eke MN ie AE UE Se 
(2.16) F = MS/MS. S73 = (1He)F)_, 


Therefore, from (2.13) and (2.16), the following relationship between 


F-statistic and op and 0, or a, and 7 can be made: 
(a) 2 pee cele 
I-l3v a 
Nae 2) 
(2097) 


[1 + (J-1) p,J aor 
1+ (J-1 ILI - 6. 
a Me 


(b) 

Feldt (1965), Nitko and Feldt (1969), Nitko (1968), and 
Cleary and Linn (1968) derived the above formula, and even applied it 
to the sampling distribution of KR20 estimates. Kristof (1963) obtained 
the same results by means of maximum likelihood methods using a multi- 
normal assumption. He obtained an estimate of intra-class correlation 
coefficient, which is equal to Py. and gave the estimate of the 
reliability of the total 6, called a step-up reliability, by using 
the general Spearman-Brown formula. However, this result is not new 
for mathematical statisticians. For example Scheffé gave similar 
results (1959, pp. 226-229). 

Because (2.17) gives the relationship between the sample 
statistic Pye or 0 and the population parameter F or p in 
terms of the well-known F-statistic, the sampling distribution of 
reliability estimates can be determined; thus, it is possible to make 
inferences about the reliability, and to calculate confidence intervals. 
Within the essentially parallel assumptions, the sampling distribu- 


tion of the reliability would not raise any questions provided the 
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assumptions (2.7) and (2.14) are all met and the model as given by 
equation (2.6) is adequate. 
As a special case of the model, let all of the fixed effects 


{85} be equal to zero, then the model (2.6) reduces to 


(2.18) ye, =u Ha, Ff e., fee WR 2 feels IT Abe’: as ane 


This model is identical to the one way random effect model ANOVA 
(Scheffé, 1959, pp. 221-235), and it can be shown that all the formulas 


given above are valid with SS, and v_ replaced by (SS, + Ss.) and 


B 
Jeol). cand MS, modified accordingly. Model (2.18) is equivalent 
to the classical parallelism assumptions (e.g., Gulliksen, 1950, p. 11) 
keen? for the distributional assumptions which are not required under 
the classical test theory. Kristof's case 2 and Maguire and Hazlett's 
case C (1969) correspond to this model 

Because the variance of random variables {a;} and tera 
are equal to on and o respectively under the ANOVA model, they 


may be rewritten in terms of standard random variables {f.} and 


fe; ,}, namely, 
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Then the model equation (2.6) becomes 
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Yeo ute S oy Ono Be 0 Eis 
= + hs . 
0 
“ei oa aaa vA re Z iJ 


or, using the notations given in Section 1.3 of Chapter One, 


i o HS 
(2-0 NG see IGE Ue yr ee 2 


with the limitations dy = do =... ry =o, ; and 


117 Yo9 = ++: = Yyy = 9, > The assumptions of (2.7) may be 


rewritten as, 


(a)? al {f.} and fe; 5} are independent random variables, 
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The distributional assumption of (2.14) becomes 
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292 Essentially Equivalent Measurements Model 


Under the ANOVA model, a = + a, + a » hence with 
Ayo) t.. - t.5, =.Cy = 16... = c. where .¢ isa constant which 
ij ij j j 


depends only on j and j'. Therefore, the part tests satisfy the 
conditons of the so-called essentially t equivalent measurements 

(Lord and Novick, 1968, p. 50) which will be denoted as ETEM hence- 
forth. Because the assumption of homogeneity of error variances is not 
required for the definition of ETEM, the error variance o may depend 
on a specific j. Thus, assumption (d) of (2.7) may be modified to 


become , 


Z 


(2.19) (d) fe; 3 are distributed with Ele; 5) =O Val (e; 5) = ee 


The variance of ai and the covariance between Vij and 


Yay: are given by 
Var (y; 5) = Ely; cE )) = 04 + ca ’ 
(2.20) 
2 
Cov (Yi poYige = E{ly; Ely; J] ly; Ey, 7} = On. 


Therefore, the reliability of jth part test is given by 


(2421) Qo. = Fc ee. 


j.e., it depends on j, and hence, in general, under the ETEM model, 
there is not a common correlation coefficient among the J parts. 


Therefore the reliability of a part cannot be interpreted as an 
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intra-class correlation coefficient. The correlation coefficient between 


j and j' depends on j and j', because, 
2 
Cov? (Feet yetey Oo 
Com (y¥.5¥.0)= ede eA ech asine  eee A 
at [var (y, )Var(y,.)]% | [(o% + 02.) (02 + o%.,) 1% 
ij i A ej' 


The reliability of the total test is given by, 
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where o, is average of ae 1? laene an (} oj’ 


If :; denotes the total varlance of jth part test given 


by (2.20), the total test variance denoted by a is 


o, = Var () Yi 3) 
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Z 2 
) o; + J(J-1) op 


Substituting this into (2,22) 
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which is the well-known formula for the Alpha coefficient. 
Novick and Lewis (1967) have shown that Alpha is equal to 
reliability po if and only if the ETEM assumption is satisfied. 


Otherwise, Alpha is, in general, lower than the reliability, namely 
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(2.23) Alpha < op 


The equality holds only if the ETEM assumption is true. 
Alpha is usually estimated by 
) Z 
Ss, 


(2.24) Alpha, = [1 =P eee 
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) ) 
Y 

where 5 and 5, are the usual sample variance of part test and the 
total test respectively. Kristof (1970) investigated the sampling 
distribution of the Alpha estimate for the case of J = 2, and showed 
that the distribution can be reduced to a Student's t-statistic by 

the maximum likelihood method. The sampling distribution for the 
general case is not yet known. 

Classically, Alpha as a reliability is derived (e.g., 
Gulliksen, 1950, p. 223) by considering two J-parts tests that are 
parallel part by part, and then introducing the assumption that the 
covariance of a part in one test with the parallel part of the second 
test is equal In the average to the covariance between any two of the 


f 
J parts within a test. If Yi denotes the score of the jth part 


of the second test the assumption is, 
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Lord and Novick (1968, p. 92) have shown that the above assumption is 
satisfied if and only if the j parts are ETEM. 

Under the ETEM assumption, the matrix model equation is the 


same as (2.6') except that the diagonal elements of ¥ may differ, 
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namely, 


All the assumptions of (2.7!) and (2.14') may be applied. 


2.5 Congeneric True Score Model 


Under the ETEM model, including the ANOVA model as a special 


4 2 ’ 
case, the true score variance, o is assumed to be common to all 


A? 
J part tests. However, for some tests, it might be more reasonable 
to expect that the true score variance would depend on j, i.e., 
some of the part tests might discriminate better and have greater 

true score variances. Under this situation, the classical parallelism 
or ETEM assumption is no longer valid. Nevertheless, the model given 
by equation (2.6') may still be used by removing the restriction of 
equal trad, namely the elements of the vector A may differ. The 
constant A. may be interpreted as a regression coefficient of Vij 


on the standard true score Fes or standard deviation of the jth 


true score. In scalar form the equation is 
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Var (AF 5) MS 
(a) ps aw, = ie? 
(2.27) J 
LET ae ea) denn ale 


where 1 is a J x 1 vector whose elements are all 1's. Since the 
ETEM assumption is not satisfied, in general, Alpha <p 

No formula is yet available for the direct estimation of the 
reliability under this model, hence the estimate of Alpha is generally 
used as an estimate of the lower bound for the reliability. Cronbach, 
feds and Avner (1964) used a similar model in their effort to 
approximate the generalizability coefficient by an intra-class 
correlation coefficient. However their model, which involves sampling 
of part tests (Type 2 sampling), assumes a uniform distribution of oe 


unlike the present model where the tA) are assumed to be fixed 


constants. Jdreskog (1968, 1970) named this model as the congeneric 


test model. 


2.4 Multi-Factor True Score Model 


The three models reviewed in the previous sections implicitly 
assumed that the test measures only one ability or trait, represented 
by f., i.e., it is assumed that the factorial structure of the true 
score is a uni-factor model. However, for certain types of tests, the 
assumption is too restrictive, and a more general model which would 


accommodate more than one true score structure is desirable. 
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If A and f, ofarevreplaced bya yd “ony! (hex F<). con- 
stant factor loading matrix A anda rx 1 standard random factor 
score vector f, respectively, the model of equation (2.6'), becomes 
the well-known multi-factor model, (e.g., Browne, 1969; JOreskog, 1970), 


namely, 

a. = 

(2.28) Ve hat Yen, a 
with, 
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Therefore, the reliabllity of the total test, po, is given by, 
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If the estimate A is available, an estimate of the reliability 


would be, 
OT 
(2.30) § = = 
S 
» 


The statistical properties of this statistic are unknown, and there 
is no agreed upon mean to obtain estimates of the factor loading 
matrix. 

Under this model Alpha is in general the lower bound for the 
reliability as with the congeneric model. The equality is true if 
and only if the parts are ETEM, i.e., r= 1. For this case the factor 


loading matrix A becomes the vector A with all elements equal, and 
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the standard deviations of the true scores are equal, as bY = Ph for 
all... j,=)1,2,...,J... lf) the error variances are all equal. among. the 

J part tests, the model becomes identical to the ANOVA model. There- 
fore the multi-factor model equation (2.28) includes the ANOVA, ETEM 


and the congeneric model as special cases. 


Under this general model the assumptions are 


(a) all {f,} and {e.} are independent random vector 
variables, 


(b) yu. =u, where up = ute, , 


(2.31) 
(c) all {f,} are identically distributed with E(f,) 


I 
k= 


D(f.) =ndb tao! E(A f.) = 0 and D(A f,) craig” 


(d) all f{e.} are identically distributed with E(e. ) 0, 


and D(e.) Si) Hor E(y ce.) = 0, and Div ec.) = y 


The normality assumption becomes 


(a) al! {f,} are distributed as N(0O, 1), or {A f.} 


are ON(Ol AN')) 


(2732) 
(b) all {e.} are distributed as N(0, 1), or {v e.} 
are N(O, vy?) 
2.5 Summary 


Four basic models which might be used for simulation of test 
scores are examined in this chapter under the assumption that a test 


has been split into J parts whose scores are continuous random variables. 


502 a i) sh ee {eve aie ‘aang ‘ 7 pve 

odd gnome: isupe~) is 246 5 ont ine notte. oft HI, helt 
“S19HT) Habart AYOMB, 9X13 oa ‘Gaohanapt! va od sae ete vanenti 
Mara: yAOMA sr4 zsbuliani (as. sy noi asus {oben 1osa8tiatum ® 
28283 Isiaaqe 26 Tabem oinemagnes. @ 


9716 @no/ sqmeee ail tsbein ‘(ersnep elas vebnll 


io 


rene eee srabnaqsbn i: 9168 on es | (33 


ae (2A 


rae 


{3 wt ‘ 


S 
jr 


. 


A) brs ge (3 


4)0 bone .0 
70. (1,0) 
10 g(t <0) 


Neild y= 


2stdsttey 
(om 


a i ola , 


Aya 


it A 


i ce 


= (aA Ol. 


‘zomaoed ne seeps yi 16 


26 batudiasaib ate is 


Can 
26 (bsdudintetb a6: dea 


— ma 


ir 


Q = ( 3 ALiw’ bedudiazeib (isoisaabi sie 1, eh) 


ca 


its 


- = es , 

Phe 
= (a, ” 
2 os (,2)3 Htiw betudiyteib ia a6 (3 te 7 y 


it «(9 one 


Pia m 
ON 


876 


ifs 


376 


7 
P — : 
- i a 
{ . 
é) 7 i i 
x 1) 
‘ 
i 
et ee \ 
‘ 
5 


38. 


The most general model is found to be the multi-factor model. The other 
three models are special cases of this model with additional assumptions 
or restrictions on the parameters. 

With uni-factor assumptions, i.e., r= 1, the congeneric 
model is the most general one, which includes the other two models as 
special cases. However, the Alpha coefficient is identical to the 
reliability of the total test score if and only if the ETEM assumption 
is satisfied. Hence under the multi-factor or congeneric model, in 
general, the Alpha coefficient is a lower bound for the reliability. 

With the homogeneity of error variance assumption the ETEM 
model becomes identical to the ANOVA model, the most restrictive one, 
and ene distribution of reliability estimate is related to an F- 
statistic. Under more general models, the distribution is in general 
unknown. 

lf equal means are assumed among the J parts, the ANOVA 
model becomes identical with the classical parallelism model except 


for the distributional assumptions. 
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CHAPTER THREE 


TEST MODEL FOR THE BINARY ITEM SCORE CASE 


For a test consisting of J binary items as the parts of 
the test, the Kuder-Richardson formula 20(KR20) has been widely 
used as a special case of the Alpha coefficient with little investi- 
gation of its statistical properties. Feldt (1965), and Cleary and 
Linn (1968) treated the discrete case as a continuous part score case. 
However, the imposition of the zero-one scoring scheme violates not 
only the assumption of continuity of part scores, but also homogeneity 
of error variances and independence of true and error scores. The 
violation of the assumptions of the ANOVA model was fully discussed by 


Feldt (1965). 


3.1 Normal Ogive Model 


To investigate the statistical properties of test scores 
of binary item tests, a number of mathematical models have been 
proposed such as the normal ogive, logistic, and binomial models. The 
first two assume existence of a latent trait or factor score f, which 
can account for the subjects behavior or performance. The binomial 
model relies on the 'strong true score! theory (Lord, 1965; Birnbaum, 
1968, pp. 508-529). In this model the conditional distribution of the 
test score for a given true score is assumed to be binomial. 

In the following, the discussion is restricted to the 
statistical properties of reliability and KR20 under the normal ogive 


model. Extensions to other models may be done in a similar way. Although 
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the multi-factor model for the binary test is also possible, for the 
sake of simplicity, only the uni-factor case will be examined. Under 
this model, the random variable representing the latent trait or factor 
scores, f, is assumed to be independently distributed as N(0,1), 

for all subjects in the population P, as under the continuous part 
test score models. It is also assumed that the response of the ith 
subject, with latent trait f = f. to each of J items, is determined 
by a hypothetical intervening random variable 7 which shall be 
called the ‘response strength variable' according to Bock and Liberman 
(1970). Since only the relative strength of Vij is of interest, 
without loss of generality, it may be assumed that ay is distributed 
with expected value zero and unlit variance, i.e., it is a standard 
random variable. In addition Le is assumed to be subject to random 
error, and if the value of 5 for the ith subject on the jth item 
exceeds a certain threshold constant specific to the item, denoted 

by ae the observed score of the subject, denoted by ai is equal 
to one, otherwise it is equal to zero. In this case the continuous 
response strength variable ij may be written as a linear congeneric 


true score model noted in Section 2.3 of Chapter Two. 
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where A. iS a constant regression coefficient specific to item j, 
J 
o.. is the standard deviation of the error scores for jth item, while 
eJ 


e.. iS a Standard random variable for errors as before. 


In vector notation, 
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WHETS, di, Pal aeeand €, are as defined for the congeneric model, except 


that the continuous part tests are replaced by dichotomous items. Also 


D(y;) = A\' + y? as before, and the diagonal elements of D(y.) are 


2 


the variances of ee and are assumed to be unity, i.e., 1 = MS a oa 


ROW pall. fo aly ts cg 
Thus model equation (3.1) may be rewritten, 


" a2) 2 
(3.3) cate rf + (1 a‘) ae 


where the standard random variables fe, 3 are assumed to be 
distributed independently as N(0,1). 
The constant A may also be interpreted as the index of 


reliability of the jth response strength variable, since 


Cor (y; jf ;) = Cov (yi jf) 


2\% 
E{[\ fF, + (1-25) e,]f 3 


=, 
J 


By definition the correlation coefficient between ing and f, is 


equal to the biserial correlation coefficient between a and fi, 
therefore, x is actually the biserial correlation between the 
latent trait variable f and observable item score She Since the 
correlation between Gt and the individual effect or the true score 
Note we ES Ae the square of SS may also be interpreted as the 


reliability of the jth response strength variable. 
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3.2 \|tem Parameters 


The jth item characteristic function Ps (f) is defined to 
be the expected value of a given ) Af} fon ‘Subject i; (e.g., 


Lord and Novick, 1968, p. 360), namely, 


(3.4) P (Ff) (2 Cae 


5 |f = f.) = Probability (x, , = dl fea the) 


Lord (1952, 1953), Lord and Novick (1968, pp. 358-394), 
Samejima (1969), Bock and Liberman (1970) and many others have investi- 
gated the item characteristic function under the normal ogive model. 

The expected value and variance of response strength variable 


aig given a fixed subject with f = fe, are given as, 


(3.5) Btye Atom t ae dF Var (yi, f = fol =e MS 


The distribution of vine for fixed f = f. is normal with expected 


value rf and variance 1 - MS or NDA fF, (1 - wo). Thus the 


probability that subject i with the latent trait f = f, will 


respond correctly to item j, as indicated by observed value 
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P.(f) = Probability (x; = 15 te- f.) 
1 aly =e f)? 
= ee ee f Ex z dy; 
=e . Z\\ ar. 
[2n(1 >! 8, ( j 
2\% 
Applying the transformation z = (Yi - AU - ri) : P (F) 
becomes 
(3.6) Ps (f) = { $(z)dz= o(-g,) 
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where $(z) and (-g) are the respective standard normal density 


and distribution functions. The value 9, (f) is given by, 


a e _ ,2)% 
7) 9; (fF) Of 8/0 r*) : 


Using generally accepted notation (e.g., Lord and Novick, 


1968) , 9, (f) may be rewritten as, 


: ; =gsae - b, ; 
(3.8) 9, (f) a. (f p> 
hence 
aes. YA ,7)% ; 
J J 
and 
bee = tere 
J Be 
(3.9) ; 
B. = cme 
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The item parameters ae and 7 have been referred to as 
item 'discrimination power! and ‘difficulty index' by Lord and Novick 
(1968. pp. 368-368). 

The difficulty of item j is defined as expected value of 


ATE namely, 


(3250) m, = Probability (x: 5 = l)r= E(x; ,) = ig P. (fF) o(f)df . 


After some algebraic manipulation, it can be shown (Lord and Novick, 


1968, p. 337) that, fit o(-B;). 
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3.3 Reliability of Binary Item Test 


Since direct decomposition of the response score Xo into 
independent true and error scores is impossible for the binary item 
scores, there is no direct way of obtaining the variance ratio of true 
score variance to total test score variance which has been defined as 
the reliability of a test. Nevertheless the population reliability 
may be obtained by resorting to the correlation method, namely, by 


calculating the correlation coefficient between 


2 


r= ) x, and x" = ) x 


where xij is the score of a hypothetical test item which is parallel 
in the classical sense to the jth item of the test. 


Then, 
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where Paix is the inter-item correlation between item j and j*, 
J < 
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o. is the variance of the jth item, and Goals the variance of the 
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total test score Xi The test variance of may be given in terms 


of inter-item correlation and item variance, namely, 


(3.13) Var (x.) = Var Q x; 3) = 3 Var (x; 5) + Le Cov (x55 9%p 0) 


2 
Sy rer ang 
lea) 
To obtain p= and a the inter-item covariance oo, o., eran must 
be evaluated in terms of the item parameters x and ma Lord and 
Novick (1968, p. 379) showed that for any two items j and j', the 


tetrachoric correlation between ve and x denoted by y..,, 


M3] 
can be expressed as the product of the two biserial correlations 
and ae by performing integration of the tri-variable distribution 


Xi pa\X%--,, and -f, namely 
VJ IJ 


ee we 
(3.14) Vgf eaudy 


It can be shown (e.g., Kendall and Stuart, 1963, p. 161 and 1967, p. 306) 
that the inter-item covariance may be expressed as an infinite power 
series of Yi! using Tchebycheff-Hermite polynomials, denoted by 


H_, (8) (Kendall and Stuart, 1963, p. 155). Then, 
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Therefore, the covariance may be calculated numerically. 
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By the results of equations (3.11), (3.13), (3.14), and 
(3.15), the reliability p, given by equation (3.12) may be evaluated 


numerically if the item parameters {854 and {a5} are specified. 


3.4 KR20 Coefficient and Its Estimate 


The Alpha coefficient for the binary item test, KR20, is 


defined as, 


j y 0? ben a) 
(3.16) KR20 = <> [1 - —H4] = sy - —4 
oO O 
x x 


which is equal to the reliability op if and only if the ETEM 
condition of (2.25) is satisfied. In general the condition is not 


satisfied, hence, 
(7) KR20 < 0p 


Using the results of equations (3.10), (3.13), (3.15), and 
(3.16), KR20 may also be evaluated numerically if the item difficulty 
{mJ and biserial correlation {a5} are specified, provided f, 
and fe; 3} are distributed independently as N(0,1). 

The ETEM ea dition of (2.25) for binary item cases in terms 


of the powe series of (3.15) may be written as, 
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This means that all the item parameters are equal. 
An estimate of KR20 is obtained by substitutiing the sample 
estimates of {n,} and a , Namely, 


més lisse :) 
(3-18) 28 = pee ee 


S 
x 


where 7 is the sample difficulty of the jth item, and a* is the 


sample variance of the test score Xe given by, 


uF = oF x AI i 
(3.19) 
of. D (x, = x,)?71/1 


Unlike the case of reliability estimates under the ANOVA 
model, the exact sampling distribution of KR20 estimates given by (3.18) 
is unknown even with the restrictive mathematical models and assumptions. 
Aoyama's formulas (1957), provided an approximation for the expected 
value and the variance of KR20 estimates without any distributional 
assumptions. He gave approximate formulas for E(KR20) and 


Var (KR20) as, 


a 
(3.20) E(KR20) = KR20 + O(1/J) , 
and 
2 Velo 2 
(3.21) Var (KR20) rays ny) (Spats oi beet aoe 
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the distribution of x; and x is the minimum score. Formula (3,20) 
indicates the estimate is biased, while (3.21) suggests a bound for 
the standard error of KR20 estimates, but is of little use since it 


involves the unknown parameter 6 which would be very difficult if 


Din 


not impossible to evaluate. 


3.5 Summary 


To examine the feasibility of simulating binary item test 
scores, the well-known normal ogive model is reviewed in terms of two 
basic item parameters, namely the item difficulty {nm 3, and the 
biserial correlation between items and the latent trait or factor 
score fe, ine@s- {a5}. 

With the help of the ‘response strength variable', a model 
equation similar to (2.26) used in the previous chapter was introduced. 

Item characteristic functions and item indices are also 
examined in terms of the two sets of parameters. 

The population reltability and KR20 were found to be amenable 
to calculation through numerical means in terms of these parameters. 


The exact sampling distribution of KR20 estimates is in general unknown. 


(o8-e), een sp103e sumtin odd i ae bre. 7 ‘hater oO NO) due i ide nae 
4? bewod 6 et2oue Us £) at It i lat ono 
$i. sept eeu alasit 0 24 aud, aatemiize, osax Yo: vores 


rhe hal 


¥1 aust ib yey od biwow atin, ig?’ “tetemei6g sivonlaw 
.si5ufave ‘ot ald 


be j 


4 


jeed mor “vient pert ey, aiTidiegs? ads on imeKe of, 
owt to amnred ni ‘beweiver 2} Tsbom vine ey nwaha= Paw, sty 


atta. bre on yituai77ib mag i edt yismen | watt 


i 
< 
: oe " 


1ot2K6T 10 Jie%3) dastet any bas ain’ nsewisd j naiteteaen | 


i i 
oben 6 - etdalney dspnense “aenogesy: atts to che arts tsiW i) va 


-beswbion int 2ew redqedo ewoiverg gt at ‘peeu (AS.8) of ae 
ols pie zoattint het: bas enotyonut olaeiveazetedy aoe 

| -eistemensa ie as int, add to einat nit 

“efidomane sd ot baw? oro OSAA- bas ynid idebl oy. noabaton ah nal 7 | 

+ | prev anetieg seeds a erin ni eb Hea tnsmid ewes sa, 
saworiants a éi zi es em to natawii Sib. anihgmex a is 
a _. +n on 


ae f 


49. 


CHAPTER FOUR 


RATIONALE FOR SIMULATION, COMPUTER PROGRAMS, AND 


METHOD OF INVESTIGATION 


4.1 Violation of ANOVA Model and Assumptions 


The review in the previous three chapters indicated that the 
exact sampling distribution of the reliability estimates is largely 
unknown except for the special case of the ANOVA model under rather 
restrictive assumptions. The distributional thoery and inferences 
based on this model are valid, if and only if all of the underlying 
assumptions of (2.7') and (2.14') are valid. A workable formula for 
the standard error of the reliability estimates may be obtained under 
this model only by employing the well-known characteristic of an 
F-statistic (Cleary and Linn, 1968). However, if any one or more of 
the assumptions are not valid the true sample reliability distribu- 
tion will not be the same as that given by (2.17). 

If the ANOVA model equation (2.6') is taken as the basic 
model, the more general models and the normal ogive model for the 
binary item test may be considered as being assumption-violating cases 
of the basic model. It has been shown that the latter more general 
models are obtained by successively relaxing some of the assumptions 
of (2.7'), and the normal ogive model has been shown to be the con- 
generic model if the hypothetical 'response strength' variable is used 
in the model equation. Thus the problem of investigating the distribu- 


tion of reliability estimates using models other than the ANOVA model 


, “a ; . : ™ ] : : pra 
by i | ‘ i ; = td ry 4 
wa : 7) ae : fa) 34 i a ye 
Pak | hot ciel ty 
i [ f 1 

y i t ' i 
4 
J 

l 


QMA eRRRORS aaTusHo3 HOFTALUNI2 ROA 
worraaiTeaMit 30 aowrak 


furan a oN ancizd pen bns Jat ol AMOWA. 26: - el oIy | 


1 


ons send iadiaten acd meets euoiverg eft pi wisi vor ait 
| Mtsgyst ot eotemitze watt difen bas to nolapatvrett an ee 
yoke yebau fsbom AVOVA eit Yo S260 Tél age. Bits 40% anes 

i zeonsietn! bine y190ns) feral aust a see odT enol Tame ae § : 
gniyl eb edt Yo Ile Ti vino bas vi ibilev S46 fsbon ver 
or ehumie? ofdejrow A »bi tev S16, (wit. sg) bas! ( v8) to sao 


Ti. 6 
ve" 


vat seas haalaa ad yaa ecdentsas 2 ff Pat ont 0 AOVIS basbr no 
ies 


ran 


| ne Yo: ielteeine awona- -Waw on gniyotans “a ytd’ | 
| to a7om %0 ‘sno yas Ti Tavswoll (838) rend: be vies) * vi 
“udiaa2ib eri edsh ras oliqmsz ound arty bitsy ial 58 wrote 
| : | eh $s) xd nsvip: gods 2 anise ody i) 308 a 
vi ‘aized sls 26 abohd ai 42, s) folteups: Vebem AVOKA ona uw ne 
my ety 40? abom evieo. Tema se bos see Intense som att ue 


oe a 
oe ont ad 03 aware need 26e | abom Svigo lemon acd? brs ee ‘Ve 


ew)? 
ri 
i ese8> golseleivenot sqnuees pniad 26 babetitoas od yem seo inst 
p Ml a , | 
daa Ieyen98 svom 19996 | of ‘feds nworte nded eed 31 stem steed eds 
i a9 7 
a f + enol sqnuze6 eo te ofige “optmaten ylaviezssoue yd banistdo oe at - 


eu ret icabe "Wpnerie sanoqesy' [63 Jldoqys ans a tabom vine 
be 3 bngnepoa te wolderg arts aunt hed deups Jobom § 
m AVOMA Ades site! ditties gis 20d emi nee aii pap 


F< e Oe a 


7 ‘a 


DOs. 


becomes the problem of investigating the effects of the violation of 
the assumptions of the ANOVA model upon the distribution of the 
reliability estimates. 

It is also suspected that, under certain circumstances for 
real data, cases arise in which the distributional assumptions of 
(2 1a)" ‘are substantially violated, that is the distribution of the 
true scores and the error scores may be skewed, and/or platykurtic 
or leptokurtic (Lord, 1960, 1969). 

However, regardless of which model the real data may 
satisfy, in practice the reliability estimates are usually obtained 
using the Alpha or KR20 formulas; hence the distributional theory 
of the estimates based on these formulas becomes a central concern 
for the test users as well as the theorist. Thus, it seems 
justifiable to investigate the distributional problem using models 
other than the ANOVA model, e.g., those in which a systematically 
distorted distribution arises for (2.17) by violating (2.7') and/or 


(2.14'). The assumptions underlying such models are summarized in 


the following table. 
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TABLE 4.1 


Summary of the Assumptions Under Various Models 


independence of 
true and error 
scores 


uni-factor 
true scores 


ETEM assumptions 


homogeneity of 
error score 
variances 


normality of 
true scores 


normality of 
error scores 


(Cong. = congeneric; M.F. - multi-factor; N.0. = normal ogive) 


puiicaie only to the response strength variable. 


4.2 Robustness Under Violation of Assumptions 


It has been known that, under certain conditions, the 
F-test of the one way fixed effects analysis of variance model is 
quite robust against the violation of the underlying assumptions. 
It may then be asked whether or not the same robustness exists for 
inferences about the reliability based on (2.17), which relies on 


an F-statistic. That is, can the findings for the one way fixed 
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effects model ANOVA be generalized to the two way mixed effects model 
ANOVA case with one observation per cell. If the sampling distribution 
of the reliability estimates is stable with the violation of assumptions, 
statistical inference based on (2.17) would be very powerful. If the 
sampling distribution of the most often used Alpha or KR20 estimates 

are found to be quite robust against the violation of the assumption 
users may freely employ the Alpha and KR20 estimate formulas and 
perform statistical inferences based on (2.17) without investigating 

the adequacy of the models or the assumptions. If the distribution is 
robust only under certain conditions, the researcher should keep this 

in mind whenever making an inference about the reliability or inter- 
preting an estimate based on (2.17). Therefore, the basic question to 
be answered is: under what conditions, if any, do the Alpha or KR20 


estimates have a stable distribution against the violation of 


assumptions. 


4.3 An Empirical Approach Toward the Problem 


Since a mathematical answer to the above problem is not 
available at the moment, and it seems impossible to give one in the 
near future, one alternative approach to be considered is the per- 
formance of an actual experiment, i.e., an empirical examination of 
the sampling distribution under various models and assumptions that 
violate the ANOVA model and its assumptions. The empirical distribu- 
tion of the Alpha and KR20 estimates can then be found and compared 
with the theoretical one under the ideal ANOVA model. 

An experiment with real data is almost impossible since the 


population parameters are seldom known. Even if this were possible 
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the data would not fit the specific model and assumptions except for 
rather limited cases (e.g., Baker, 1962). One available method is to 
use computer simulated data, under various assumptions, to obtain 
empirical distributions of the Alpha and KR20 estimates and compare 
them with the distribution for the ideal ANOVA model. 

The author has already investigated the feasibility of such 
computer simulation techniques in the study of the effects of the 
violation of assumptions on the F-test for linear models requiring 
statistical inferences, and has provided a comprehensive computer 
program for educational and psychological researchers (Bay, 1970). 

The present study uses essentially the same techniques to 


investigate the sampling distribution of reliability estimates. 


4.4 The Concept of Simulation 


The term 'siimulation' has been used rather uncritically in 
a wide range of scientific or economic fields, especially for the 
purpose of building models. Von Neuman and Ulam's work in the late 
1940's, when they attempted to solve certain nuclear physics problems 
by a Monte Carlo analysis, may be considered the first modern use of 
the simulation techniques. A Monte Carlo analysis involves the solu- 
tion of a problem, that is either too expensive for experimental 
solution or too complicated for analytical methods, by simulating a 
stochastic process that has probability distributions satisfying the 
mathematical or probabilfstic relations underlying the problems. 

With the development of high speed computers in the last 


two decades, not only physicists and other natural scientists, but also 
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economists, psychologists, and other social scientists can perform 
controlled laboratory-like experiments on a computer with much 
efficiericy and economy. 

Although there is no agreed upon definition of the term 
‘simulation', for the purpose of this paper it was considered sufficient 
to use the following definition given by Churchman (1963, p. 12). 

'x simulates y' is true if and only if: 

(a) x and y are formal systems, 

(b) y is taken to be the real system, 

(c) x is taken to be an approximation to the real system, and 


(d) the rules of validity in x are non-error free. 


In the context of this paper, y is a system which produces 
a number of real test score sets by performing actual random sampling 
of subjects and administering the test, thus giving a number of real 
estimates of reliability of the test. The number of estimates under 
the real situation is limited since the actual sampling of subjects 
and the administration of the test are involved. On the other hand, 
x is a system which produces a number of test score sets, via computer, 
under a model and a number of assumptions which will approximate the 
real system y. Since the number of test score sets obtainable under 
x is almost unlimited, the sampling distribution of the relaibility 
estimates is easily obtainable by calculating the frequency distribu- 
tion of the estimates. Furthermore, since the test parameters and the 
distributions of true and error scores can be manipulated easily under 
computer simulation, almost any combination of models and assumptions 


can be investigated. The researcher can input the most appropriate model 
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and assumptions which will best approximate the real system y fora 
given test and population of subjects. 

This approach toward statistical inference is somewhat different 
from the conventional procedure since the user can choose the model and 
assumptions of interest to him, while in the conventional case the model 
and assumptions are predetermined by the mathematical statisticians and 
the user can only choose whether or not to accept the conditions and 
the model, look for alternatives, or give up. In this sense computer 
simulation techniques permit the study of sampling distributions under 
almost unlimited combinations of models and assumptions. Thus, the 
user may obtain the sampling distribution of a statistic under his own 
pede and assumptions in the experimental situation, make statistical 
inferences, and use the knowledge so gained in practice. Because of 
the fourth property of the simulation, the method may not provide 
exact answers, but it would provide approximate answers to distribution 


problems. 


4.5 Computer Programs 


Two computer programs named RELOI] and RELO2 have been 
developed in FORTRAN IV on the IBM 360/67 computer of the University 
of Alberta computer system for continuous part test and binary item 
test cases respectively. The programs are in sufficient general form 
so that they can be used for other problems related to sampling 
distribution of reliability estimates not considered part of the study. 
The programs automatically simulate the test score matrix Y= tyiat 
for the continuous case, or X = {x) 53 for the binary case based on 


input models, parameters, and specified distributions of true or latent 
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scores and error scores. The programs have the following features: 

(a) For the continuous case, the program RELOI] uses the 
most general model, namely the multi-factor true score model given by 
(2.18), and accommodates all other less general models as special 
cases. Users are able to specify the sample size |, the number of 
parts J, and the parameter vector and matrices for the model, namely 
u, A, and ¥, i.e., mean vector, factor loading matrix, and error 
standard deviation matrix respectively. The program will evaluate 
population test parameters such as reliability, Alpha, mean, true 
and error variances. 

(b) For the binary item case, the program RELO2 uses the 
normal ogive mode! (3.1), under a uni-factor latent scores assumption, 
and allows the user to specify the sample size |, the number of items 
J, and the basic item parameters, namely the difficulty parameters 
{n,} and the biserial correlations {aj}. The program will evaluate 
the population test parameters such as pp, KR20, and o based on 
the Tchebycheff-Hermite polynomials discussed in Chapter Three and 
other formulas under the normal ogive model. However, these cal- 
culations are valid only for the normal distributions of latent 
scores and errors. If the normality is violated, the parameters 
calculated are no longer valid, unlike the case of continuous parts 
where the test parameters are independent of distributions of true 
and error scores. To evaluate test parameters for non-normal cases, 
an empirical method based on a parallel form method described in the 


following Section 4.6 is used. 


(c) For both programs the user may decide shapes of the 
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distributions of true or latent scores {f,} and error scores te, ,}. 

The programs generate speclfic distributions by means of random number 
generating subroutines. The distributions of true or latent scores 

and error scores are specified by user supplied subroutines DIST and . 

DISE respectively for non-normal cases. These two subroutines may call 
the uniform random number generating subroutine VECRAN described in the 
following Section 4.7. For the normal case, the program generates the 
distribution automatically by employing the Box-Muller method which is 
also described in the Section 4.7. 

(d) The programs automatically perform N simulations, as 
specified by the user, and calculate a number of test statistics. The 
reliability coefficients are estimated for each simulated test score 
matrix based on the formula (2.13), regardless of the model and 
distributions used to generate the score matrix, since the formula is, 
as was noted before, the one most often used by the test theorists or 
users. Alternatively or concurrently, as an option, the user may 
adopt an unbiased estimation formula developed by Kristoff (1963) and 
discussed in the following chapter. The distributions of reliability 
thus estimated are then compared with those obtainable from (2.17), 
i.e., the ideal ANOVA model and normal theory. For non-normal binary 
item test cases, the reliability parameter obtained by the parallel 
form method is used for the value of op. 

(e) The programs also summarize the empirical distributions 
of MS), MS. MS.) and 6 by calculating their means and variances 
over N samples, and compares them with the theortical values of the 


expected mean and variance under the ANOVA model and normal theory 
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assumptions. For the binary item case, the variance parameter on 
and 0 in terms of the test score X; are not defined or calculable 
directly. However, from the definition of reliability given by (2.12) 
and the variance given by (2.10), a generalization of the relationships 


between reliability and variances to binary item cases may be made such 


that the formulas in Chapter Two may be used without modification, 


namely, 
2 2 
(4.1) a gi : ve Clap soa 
On 2 ’ oe a] > or 
nape 2 *, #2 
(4.1') ig AB a a oe Oepaias 
eat On 7 : z 5 ; 


for non-normal cases, where the star (*) notation referes to para- 
meters evaluated by the parallel form method. 

(f) Comparisons between the empirical distributions of 
reliability estimates based on either or both (2.13) or Kristof's 
unbiased formula, with those theoretical ones based on (2.17) or 
modified form of it for the unbiased formula, can also be made as an 
option by plotting both distributional curves together in a graph. 

Computer program listings together with example outputs 


of the programs are given in Appendix A.1 and A.2 respectively. 


4.6 Parallel Forms Method for Test Parameters of Binary Item Test 


For the continuous part test cases, test parameters such as 


o-, oe, and Alpha depend only on the input of part test parameters 


and are independent of the distributions of the true and error scores. 
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However, for the binary item test cases, the basic test parameters 
depend not only on item parameters such as difficulty or biserial 
correlations but, also on the distributions of latent scores and 
errors, since the normal ogive model connects the continuous response 
strength variable ri to the binary item score eT Therefore 


a ep, and KR20 given 


the formulas for test parameters such as a 
in Chapter Three are valid only for the case of normal distributions, 
In order to be able to investigate the sampling distributions of 
reliability estimates under non-normal cases, i.e., under the assumption 
violating cases of the normal ogive model, the test parameters must be 
known by means other than these formulas. Although for some simpler 
distributions such as the uniform distribution, evaluation of these 
parameters by analytical means might be possible, a general solution 
to cover all types of possible distributions is impossible, and 
alternative empirical methods are employed in the RELO2 program. 

Since the number (N) of test score matrices simulated is 
usually large, say at least 1000, the number of test scores (Nx 1!) 
simulated in each experiment is a very large number compared with the 
sample size |. On the other hand, the sample reliability and variance 
are consistent estimators of the corresponding population values. There- 
fore, if Nx | test score sets are used at a time to estimate these 
parameters, the estimates will be close to the population values. How- 
ever, to obtain a population reliability coefficient by this large 
sample method and the correlation formula give by (3.12), parallel 
form test scores must also be simulated which have identical f; 


terms but different Si terms denoted by ej due to random 
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fluctuation of responses. Therefore, two sets of model equations may 


be considered for the response strength variables Yije namely, 


2\4 
. = ef. + ee tee fie 
hore Hiikietbubbon:*1 tne bis 
(4.2) 
* 2y5 * ; 
ee = > < + i . a oe = . ° = 
Vij rf (1 r) ee | rae e bem Pal hte et 


Form these model equations, two sets of test scores {x, } and {xi} 
may be generated, and by calculating the correlation coefficient 
between these two sets of scores, the population reliability may be 
obtained regardless of which distribution is used for simulating the 
test scores. For ideal normal cases, the parameters obtained by this 
method should agree closely with the calculated values based on the 
formulas of Chapter Three, providing one way of checking the formulas 
in the chapter and the computing procedures adopted by RELO2. The 
population parameters thus estimated will be denoted by the correspond- 
ing population parameter symbols with a star (*) sign to distinguish 
them from those obtained by analytical means. For example, the test 


mean and variance obtained by this method are given by, 


ae *2 Be PD 
u = () x,)/NI, and fog t) cere ta) NI 

i i 
4.7 Procedures for Generating Random Numbers 


The method of generating a set of independent random numbers 


with a specific distribution by a computer program is of extreme 


importance to the success of a stochastic simulation experiment. The 
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simplest and basic set of random numbers with a continuous probability 
density function is the one that is constant over the interval (0,1) 
and is zero otherwise. The density function defines what is known as 
a uniform or square distribution. The principal value of the uniform 
distribution for the simulation techniques lies in its simplicity and 
in the fact that it can be used to simulate random variables from 
almost any kind of probability, distribution since the inverse trans- 
formation of the cummulative distribution function of any random 


variables results in the uniform distribution between (0,1). 
The uniform density function on (0,1) is defined by 


(4.3) f(z)=) 1.0 0O<2<)| 


otherwise. 


I 
oO 
(2) 


Due to its simple density function, it is very easy to 
evaluate moments for such a uniformly distributed random variable 
by using elementary calculus. 

For this study, the method used for generating uniform 
random number is the same as that used by the IBM Scientific Sub- 
routine Package RANDU (IBM, 1968). The subroutine named VECRAN 
can however generate a specified number of uniform random numbers 
at a time and provides the output in vector form, while only one 
number at a time is generated by RANDU. 

The method employed is the so-called 'power residue 
method', (IBM, 1959) or ‘multiplicative congruential method' (Nayler 
etal, 1968, °p. 51-52). The method generates successive non- 
negative integer random number which are less than 2 for binary 


computers where c denotes the word size of the computer by means 
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of a congruence relation, namely, 


(4.4) Mp ea. (moduil.gtor 2), 91 ‘= O81 see 


where N is the so-called seed random number denoted by IX in the 
program. Meanings of 'power residue', 'congruential' or 'modulo' are 
given by Nayler et al., (1968, pp. 63-66), or can be found in any text- 
book of elementary number theory. The formula (4.4) is the so-called 


/(2°-1) 


formula for generating power residuals, and results in u = ney 
being approximately a uniform random number in (0,1). For the 
[BM 360 series computers, c = 3], amd VECRAN uses a = 65539, 


mat 


BPMOS = OPN6E6613e 10k?” which rel chese@anenae for RANDU. The 


user must specify Ng = IX as an input parameter at the beginning of 
the program execution, and it must be an odd integer with nine digits 
or less. The last value of n generated may be used as an input 
seed random number IX for the next step generation. 

The random numbers thus generated are often referred as 
pseudo-random numbers, and the method involves the generating procedure 
by ‘indefinitely continued transformation of a group of arbitrarily 


chosen numbers! (Tocher, 1954, p. 41). The term has been defined by 


Lehmer (1951) as, 


. a vague notion embodying the idea of a sequence in 
which each term is unpredictable to the uninitiated and 
whose digits pass a certain number of tests, traditional 
with statisticians and depending somewhat on the use to 
which the sequence is to be put. 


Although there are some objections on the philosophical grounds that 
the sequence is generated by a deterministic rule of (4.4), use of 
such pseudo-random numbers can be defended by pragmatic reason that 


a sequence may be regarded random if it satisfies some predetermined 
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Statistical tests of randomness, and the uniform number generated by 
RANDU has been known to satisfy these requirements (IBM, 1968). 

Based on the uniform random numbers thus generated by 
VECRAN, denoted by Ul, five other kinds of random numbers are 
generated for this study. For the selection of these specific types 
of a random number the following factors were taken into account: 

(a) Ease of generation and computer time required for 
computation. 

(b) Ease of evaluating the moments of random numbers by 
calculus to ensure that the program generates random numberswith the 
required distribution. 

(c) Some practical usefulness. For example, normal, 
uniform, and exponential distributions are included because the 
approximation of the normal distribution to real data is so often 
assumed, the uniform distribution is closely associated with ranked 
data, and the exponential distribution can arise with the truncated 


data of normal distribution due to a selection process. 


The six kinds of random numbers, including Ul, are 


summarized in the following table. 
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TABLE 4.2 


Summary of the Random Numbers 


Description Transformation Formula 


Uniform, ,: (0.1) Ul z= u 
Sum of 2 indep. UI U2 z= {(ujtu,) - 1.0} x (6)% 
Sum of 3 indep. UI U3 z= { (uj +u,tu,) = Pooh ene 
Sum of 6 indep. UI U6 z= {(u,tuy+...tug) - 3.0} x (2)% 
Normal NO a (-2 Ln u)? Cos (2nu,) 

AS (-2 Ln u)? Sin (2nu,) 
Exponential EX z = -Ln (u,) also 


Note: Upsees Ug denote the uniform random numbers generated by 


VECRAN. 


The method used for the generation of standard normal random 
variables is the same as given by Box and Muller (1959). Since the 
distribution is exact, it has an advantage over the so-called central 
limit approach which uses the sum of a number of independent uniform 
random variables. All random variables in this study were used in 
standard form, namely with an expected value of zero and unit variance, 
except Ul which was standardized by subtracting 0.5 and multiplying 
by the square root of 12.0. Therefore the random numbers thus 


generated can easily be used as if} or fe; ;} of the model equations 


(2e23)0 (303), and (4.2). 
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A number of preliminary sampling experiments were performed 
to ensure that this method generates the random numbers with desired 
distributions. To see whether the means, variances and other statistics 
for large samples closely approximated the population values of the 
distribution simulated by the random numbers, five samples of size 
6000 each were generated for each distribution, and the obtained 
statistics were compared with the population values obtained from the 
knowledge of the probability density functions and the application 
of elementary calculus. The results are summarized in Table 4.3, 
With some exceptions for the calculated kurtosis of the distribution 
noted with (*) sign, the sample statistics approximate reasonably 
Belli the population values. The exceptional cases are probably due 
to the imperfections of the random number generating procedures and 
sensitivity of kurtosis to the shapes of the distribution. The 
calculated auto-correlations are almost zero indicating no serial 
correlations for adjacent random numbers in the sequences and the 


degree of independence of random numbers thus generated. 


4.8 Methodological Limitations 

Because the computer simulated experiments cannot be 
exhaustive and cover all possible combinations of models, parameter 
sets, and distributional assumptions, and due to the very nature 
of computer simulation techniques and limited funds available for 


the computing charges, the following methodological limitations are 


imposed on this study. 
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TABLE 4.3 


Descriptive Summary of Random Numbers Generated by Pseudo-Random Number 
Generating Subroutines, Sample Size = 6000 for Each Trial 


Skewness Kurtosis 


5 
Expected 


Expected 


Expected 


Expected 


1 
2 
3 
i 


5 
Expected 


Expected 


All random numbers are standardized 


Auto-Correlations 


Lag | Lag 2 


21110 
azZalsi 
- 20730 
e19213 
18213 
- 20000 


58865 
. 59803 
63365 
-58109 
.65140 
.60000 


42060 
30457 
37746 
. 37966 
40427 
- 40000 


23413 
. 20790 
.2206) 
. 24665 
16114 
. 20000 


-11501* 
-05017 
-03248 
.04551 
.05775 
.00000 


.70110 
31563 
.02954* 
9161 4% 
. 10958 
.00000 


by population mean and variance except UI. 


Lag 3 
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(a) The investigation is restricted to the sampling distribu- 
tions of reliability estimates under Type 1 sampling situation only, 
namely only sampling of subjects iS involved; the test is assumed to 
be given and all parameters for part-tests or items are assumed to be 
fixed constants. The distributions under the Type 2 or Type 12 sampl- 
ing situation, such as the distributions of generalizability coefficient 
estimates, are not considered in this study, although this may be done 
very easily as an extension to this study. 

(b) Because the computer time required for each experiment 
must be kept within reasonable limits, the sample size |, the number 
of parts or items J, and the number of samples to be simulated must be 
fepe within moderate bounds for this study. Therefore, although the 
programs are dimensioned such that they can accommodate up to 
N = 5000, | = 100, J = 30, investigations are limited to N = 2000 
or 1000, It = 30, J = 6, 8, or 9 to restrict each experiment within 
5 to 7 minutes of C.P.U. time which costs approximately $20-30 at the 
present charging rate of the University of Alberta. 

(c) To conserve computer time for the overall study, the 
experiments have focused only on the following key problems: 

(i) The effect of non-normal true or latent scores 
and error scores distributions. 
(ii) The effects of non-homogeneous error variances, 
j.e., distributions under ETEM model. 
(iii) The effect of congeneric and multi-factor true 


score model. 
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(iv) The effect of binary item scores, non-homogeneous 
difficulty parameters and biserial correlations, 


and non-normal distributions. 


(d) The non-normal distributions used in this study are 
limited to a minimal number of well known distributions outlined in 
Section 4.7. 

Because of these limitations, the findings of this study 
will be limited to some extent in their generalization to all ‘'real' 


situations. 


4.9 Accuracy of Calculation 


Like any other numerical analysis, the results reported in 
this study are subject to certain computational errors. The figures 
reported in this study retain, in most cases, three decimal places, 
but they may be inaccurate in the right most one significant digit 
due to the cummulative effects of errors when the sample size N_ is 
large. This is especially true for the case when the variance of a 
variable is small in comparison with the mean. However, it is expected 
that the errors are confined only within 3 to 4% level at maximum, 


and they would not affect the findings of this study. 


4.10 Summary 

In this chapter, the rationale for investigating the samp|!- 
ing distributions of reliability estimates as assumption violating 
cases of the well known ANOVA model and normal distributional theory, 


and using the computer simulation technique to investigate such 
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problems were discussed. The computer programs developed for this 
study were outlined, and the parallel form method, the random number 
generating procedures, and the methodological limitations due to the 


very nature of computer simulation techniques were also discussed. 


TD, 


CHAPTER FIVE 


RESULTS FOR CONTINUOUS PART TEST SCORE CASES 


This chapter presents the results of the computer simulated 
experiments for the continuous part test score cases. Section 5.1.0 
deals with the effects of non-normality under the ANOVA model; and some 
analytical methods are also used to investigate the standard error of 
reliability estimates. The distributions of reliability estimates under the 
ETEM model are dealt with in Section 5.2.0, and in Section 5.3.0 for the 


congeneric and multi-factor true score cases, i.e., non-ETEM cases. 


Bai l0 Effects of Non-Normality Under the ANOVA Model 


5.1.1 Distribution Under ANOVA and Normal Distribution of True and 
Error Scores 
It has been shown in Chapter Two that, under the ANOVA model 
and normal distribution, the reliability estimate given by (2.13)-(b) 
can be related to an F-statIstic by the equation (2.17), and it can 
also be shown that (Kendall and Stuart, 1963, p. 393), 


te 2 Ae tne) 
MT ve ilo 2 la ate) 


n 
E(F ) alga a Var (F 


Therefore, using the relation as pe = Set it Is easy to show that, 


i-1 
j= Is (I-o) >= 


(ame (6 )o=) 1) slime) Ee 


Veo 
err) 
2 wcll ola led ) 


= (I-p) 
‘ogi aaa Cis es) 


waite esugno> afe vo eviveor ity ‘ontk, i cll in 
y | more | 
ae if on. e noi ssae 29269 oxdbe gen s18q evourt Finds sat vet 2 om 


_ 
v 


: “amie boe :Tetom AVOWA 93 vebru ai Temnan=Hin to atobiis ats ftw a 

| he! 10779 bisbate ga! saegigeaval of) eee eels S16 een iz 

hie rere aosemnise9 yitlidet ley to enoi dudi+d21b ear addomi tes | 

sick 70% 0. €.¢ neitese ni bas Oh g, ai noida? a A hee Head a6 
ab262 M3T3-qon . 9.1, 292889 97002 sunt vodoet~ isu bn 


\aboM: _ edt ene Mite 


Nol 


fabom AYOMA erly isbriv 3683 ow? istnad9 a} nw naad. dda a 
(a> (et. $) yd navic ation Witidet to | ily. nT aua AND | 
a “me ai toe, (X18) nolteups atta va sitelsane-a his as boreten » 
i (ERE “4 E881 (Tysut2: bia chon) 6A wail 


é Se = er a = . a 


“hes woe 0 1 een et at P alia ah noltéelsy alt pntiew aIghe 


I 


& , 


# (on - “te Ga? fori =» aa ro 
milla s Oss Men = taps ar 


dA 


Hence 6 is in general a biased, but consistent estimator and 


does not have the minimum variance property. Kristof (1963) modified 
formula (2.13) to obtain the unbiased estimator , and has shown that 
it has a smaller variance than 6 , namely 
Ne Ear: 1-3 
he) faith Se TeqeeT TenanPs™ Tay toga Mir iSeH Sly! OF 


(52) 
leg abep) 
tb), Fang the WREST GEA) 
It can then be easily shown that 


E(6) =»; Var (6) = [417 Var (6) = (1-p)? 2tvet 3) < var (pie 


Therefore, if the equation (2.6) is the appropriate model for 
the data and the assumptions (2.7) and (2.14) are all satisfied, the 
results of equations (2.17), (5.1), or (5.2) can be used to make 
inferences about po and to calculate the standard error of estimation 


which is defined as the square root of the variance of 6. 


5.1.2 Known Effects of Non-Normality Under ANOVA 


As it has been seen, the sampling theory and the formula for 
the standard error of estimation rely heavily on the normal distribution 
assumptions, despite the fact that real data seldom satisfy these 
assumptions, and at best may be expected to only approximately satisfy 
them. It does not logically follow, of course, that approximate 
satisfaction of the normal distribution assumptions by true and error 
scores will guarantee automatic approximation of the actual distribution 


of reliability estimates to the distribution given under normal theory. 
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Scheffé (1959, p. 345) investigated the effect of non- 
normality from an analytical point of view and concluded 'Non-normal ity 
has little effect on inferences about means but serious effects on 
inferences about variances of random effects whose kurtosis Yo differs 
from zero'. He also noted that 'The direction of the effect is such 
that for confidence coefficients 1-a and significance level a _ the 


true a will be less than the nominal a if the 


YO.A <0, and greater 
if ee Vid 0, and the magnitude of the effect increases with the 
magnitude of Yo ne Although his argument is based on the inference 


of the so-called signal-noise ratio 06 = alam under the one way 
random effects model, it is suggestive for reliability theory, and 
provides a guideline for the investigation of the effects of non- 


normality under the ANOVA model. 


5.1.3 Stiandard Error of Reliability Estimates Corrected for Non-Normality 


The standard error of reliability estimates is a useful measure 
of the precision of the estimates, although, as noted in Chapter One, 
without any knowledge of the shape of the sampling distributions of the 
estimates it has little inferential use. Since reliability has been 
historically identified as a correlation coefficient, the wel1l-known 
standard error of correlation coefficient estimates has been frequently 


used (e.g., Jackson and Ferguson, 1941), namely, 


(5.3) Var (6) = —o— 


in which the assumption of bivariate normality is made (Kendall and 
Stuart, 1963, p. 236). However, this formula or those given by 


equations (5.1) and (5.2) would be misleading if normality cannot be 
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assumed. General distributional theory under non-normal true and error 
scores is not yet known, but the Var (6) or its Square root, denoted 
by S.E. (6), may be evaluated approximately if the kurtosis of the 

true and error scores, denoted by Yr and he respectively, are known 


or can be estimated. In this case 


= [E(at)/o/] - 3, 
(5.4) 
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= [e(e;,)/o¢] = #3, 


Tukey (1956) obtained the variance of the variance estimates 
under various ANOVA models by employing 'polykays'. For the model 


considered in this paper, he has shown that 
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(c) Cov (MS, » MS ) = WD? 


It is noted that if the true and error scores are normal, 


j.e., the kurtosis is equal to zero, the results are the same as expected 
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under normal theory obtainable from equation (2.15) and the resulting 


independence of MS, and MS. 


A 
Letting x» = MS,> and x, a MS. and W(x, 5x.) a function 


of x, and X5» an approximation formula (e.g., Scheffé, 1959, p. 230) 


may be applied to approximate Var (6) from (2.13)-(b) namely, 
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Var [W(x, x5) ] mee 


2 
Var (x, ) + 2WW, Cov (x), x.) + W 


> Var (x, ) ; 


where Wr denotes OW/ dx, evaluated at ee E(x,) = J of + of ,» and 
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x» = E(x.) = Ge Then vars (6) = Var l= MS/MS, ) = Var. (x5/x,), 
2 2 2x2 
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(5.7) Var (6) ~ (I-p)* I 


Formula (5.7) does not agree exactly with formula (5.1)-(b) when 
the distributions are normal since an approximation has been employed. 
However, formula (5.7) is suggestive for correction terms to be added 
to formula (5.1)-(b) for non-normal distributions, i.e., Var (6) may 


be obtained by a new formula combining (5.1) and (5.7) as 
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Since this formula involves two unestimable parameters on and Ye? 


further approximation is necessary to make it useful. 


The kurtosis of the test scores y, = ) Liaie Ju+Ja, + 
j 
) €e denoted by y., is an estimable parameter, and may be 
ley? y 


evaluated by considering it as a linear combination of J+1 independent 
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formula given by Scheffé (1959, p. 332), namely, 
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2 
Then, rl Fs Po hoy forzeps>: I,heorancy ~ 0, or J fairly large. 


Therefore, it may be shown that, 
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(5.10) Var. (6) = Chap, 5 i 
(Joay (le 3)hads) 


This formula (5.10) is, to the author's knowledge, a new one for test 
theory, which only includes the known constants 1,J and the unknown 
but estimable parameters op and Re As a result it can be used to 
obtain an estimate of the standard error of reliability estimates, 


namely, 
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(5.11) it pas = (eee (1-6) 5 
(J-1) (1-3)*(1-5) oY 


From the formula (5.8) it may be observed that the effects 
of non-normality on the standard error of reliability estimates depend 
on the following: 

(a) The kurtosis of the true scores multiplied by the factor 
1/1, and of the error scores multiplied by a factor of I/IJ. 
Therefore, the effect of non-normality would be dominated by the 
kurtosis of true scores which is closely approximated by the kurtosis 
of the test scores divided by the square of the reliability. 

(b) The magnitude of p, namely, the larger the value of op, 


the greater is the effect of non-normality. 
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The above observations suggest that the sampling distribution 
would be robust against the violation of normality assumptions if (a) 
the sample size |! is large, (b) reliability is close to zero, or 
(c) J is fairly large and the true score kurtosis (or the test score 
kurtosis) is close to zero. The condition (a) is of little practical 
value since statistical inference problems usually arise for the smal] 
sample case, while (b) is also of little practical value since, in most 
cases, reliability theory deals with p close to unity rather than 
zero. The last condition indicates that the sampling distribution of 
reliability estimates would be robust against the violation of normality 
of errors for J fairly large, and is sensitive to the distribution 


of true scores. 


5.1.4 Results of Simulation Experiments Under ANOVA Model 


In order to investigate the effect of non-normality under 
the ANOVA model, a number of experiments were performed by RELOI] using 
the following distribution-parameters combinations with the constants 
N = 2000, | = 30, and J = 8. 

(a) For the distribution of true scores, all of the six distri- 
butions discussed in Table 4.2 of Chapter Four, namely Ul, U2, U3, 
U6, NO, and EX. were used. 

(b) For the error scores distributions, the uniform, normal, and 
exponential distributions were used, i.e., Ul, NO, and EX respectively. 

(c) Three levels of p were used by fixing 0” = 4.0, and using 
ERPSE favel's “OF on namely, 4.0, 1.0, and 0.36 to indicate high, 


middle and lower levels of reliability. 


Altogether 6 x 3 x 3 = 54 experiments were performed, each 
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requiring approximately six minutes of C.P.U. time. Since the parameters 


and {8 53 do not affect the distributions, they are not reported. 


In Table 5.1, the observed means and variances of MS and 
MS. for N = 2000 samples are presented with the theoretical values 
based on formula (5.6). Because formula (5.6) does not involve any 
approximation, any disagreement between the observed and calculated 
values must be attributed to either sampling fluctuations due to the 
finiteness of N or deficiencies in random number generating methods. 
It is noted that a rather close agreement exists between the observed 
means of MS and MS. given in columns (1) and (3) with their 
theoretical expected values given in column (7). Comparisons of the 
observed variances of the MS's given in columns (2) and (4) with 
the theoretical values based on (5.6) given in columns (5) and (6) 
Suggest that the two agree reasonably well, although the agreement is 
not as close as that for the means and expected values, which probably 
reflects the imperfectness of the random number generating procedures 


and/or the sensitivity of the variance to the change in the shape of 


population distributions. 


Column (1) of Table 5.2 contains the mean of 6 over the 
N samples. These values can be compared with the expected values 
under normal distribution theory given in column (6). It is observed 
that, for negative Y,, the means are in general higher than E(p) 
based on formula (5.1)-(a), thus causing some moderating in the 
tendency to underestimate the reliability under normal thoery. If 
Y, is positive, the mean of 6 is in general lower than E(p¢) and 


exaggerates the tendency of underestimation. The effect of ‘@s is 
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TABLE 5.1 


Comparisons of Observed Means and Varlances of MS's Under ANOVA Model and 
Varlous CombInations of True and Error Score Distributtons 
With the Values Obtainable From Formula (5.6), 
N= 2000, | = 30, J = 8 
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TABLE 5.2 


Compartsons of Observed Means and Standard Errors of Rellability Estimates 
Under ANOVA Model and Varlous Combinatlons of True and Error Score 
Distributions With the Values Obtalnable From Formulas (5.3), 

(5.1)-(b), and (5.10), N= 2000, V= 30, J=8 
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almost negligible, as expected from the discussion of the standard 
errors in Section 5.1.3. Therefore, as far as point estimation is 
concerned, negative kurtosis would not cause any serious problems, 


but positive kurtosis may cause serious underestimation of reliability. 


Table 5.2 also contains the standard errors of 6 in columns 
(3), (4), and (5) under various formulas as well as the observed results 
in column (2). The results clearly indicate the inappropriateness of 
the tranditional formula (5.3) or the more recent formula (5.1)-(b) when 
Y, iS non-zero, and demonstrates the effectiveness of formula (5.10). 
To see how closely the values based on these formulas approximate the 
observed values, the sum of squares of the deviation from the observed 
values are calculated with the results 0.0416, 0.0133, and 0.0013 
for formulas (5.3), (5.1)-(b) and (5.10) respectively, with the 


minimum deviations for formula (5.10). 


To examine the robustness of the F-test based on formula 
(2.17), under normal distribution theory, the shapes of the upper 
and lower 5% tail portions of the distributions of 6 were investigated. 
Columns (1) and (2) of Table 5.3 show approximate real Type one errors 
when nominal significance levels are fixed at 5% level for each tail. 
The results clearly indicate that real Type one errors are less than 
the nominal value If Yq is negative, and the smaller is Yq the 
smaller is the resulting real Type one error. For positive Ya? the 
real Type one errors are greater than the nominal value. These results 
are in close agreement with the Scheffé's conclusion referred to in 
Section 5.1.2. It is also noticed that the effect of non-zero T is 


less for small p, i.e., the test is robust if po tends to zero as 
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TABLE 5.3 81 . 


Comparisons of Observed Lower and Upper 5% Critical Polnts of Rellabl lity Estimates Under 
the ANOVA Model Using Varlous Combinations of True and Error Score Distributions, and 
Real Type One Errors of F-lest When Nominal Value Is 5% With the Values 
Obtalnable Under the Normal lheory, N= 2000, 1 = 30, J 8 
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anticipated by the earlier discussion of the standard errors of estimation 


im section 5-1. 3. 


5.1.5 Conclusions on the Effects of Non-Normality Under ANOVA Model 


From the above discussions the following conclusions are 
tentatively made: 

(a) The effect of non-normality of the error score distribution 
is negligible for J fairly large, where J is the number of part-tests. 

(b) Non-zero kurtosis of the true score distribution substantially 
effects the sampling distribution and standard error of reliability 
estimates. 

(c) The F-test under normal theory is robust for near zero 
population reliability, or near zero true score kurtosis, if J_ is 
fairly large. 

(d) Formula (5.10) is superior to the traditional formula (5.3) 
or (5.1)-(b) for the calculation of the standard error of reliability 
estimates. 

(e) For the F-test, the real Type one error is lower than the 
nominal value for negative kurtosis, and higher for positive kurtosis 
of true scores. This true score kurtosis is closely approximated by 


test score kurtosis divided by the square of the reliability. 

The above findings are restricted to the ANOVA model, and 
generalization to more liberal test score models requires further 
study. 


5.2.0 Relaxation of the Homogeneity of Error Variance Constraint in 


the ANOVA Model 
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5.2.1 The ETEM Model 


For the ANOVA model it was assumed that the 

variances of error scores te, ,} were homogeneous, f.e., all the error 
variances {o,,} are equal to an unknown constant on by assumption 
(2.7)-(d). This assumption was made not because real data are expected 
to have homogeneous error variances, but to make the mathematical 
abstraction simpler. Therefore it is conceivable that the error 
variances may differ for each part test, i.e., for real data the 
variance of Fj may depend on the part test j, as given by (2.19). 
Under this last assumption, the model becomes an essentially Tt 
equivalent measurement (ETEM) which was discussed more fully in Chapter 
Two. Under this model, there is not a common intra-class correlation 
among the J part-scores to be interpreted as the reliability of a part- 
test under the ANOVA model. But the reliability is still equal to the 
Alpha coefficient. The only difference from the ANOVA model is the 
replacement of 0 in the reliability formula (2.12) by the mean of 

Z 2 


Te denoted by o, . 


{o 
Because assumption (2.7)-(d) is violated, the distribution 

of reliability estimates given by (2.17) cannot be expected to hold 

for the ETEM models; at best it is hoped that the distribution is 


closely approximated or the distribution is robust against the 


violation of the assumption of homogeneity of error variances. 
5.2.2 Effects of Non-Homogeneous Error Variances Assuming Normal 
Distribution 


The general distributional theory of reliability estimates 


under the ETEM model with the normal assumption is not yet known except 
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for the case of J = 2. Kristof (1970) has shown theteafor J =;2. 


the statistic 


= s ak 
(S212) t= a Ee ee ey 
6(I-p)? (s+ s2 - 52.) 
gly LZ 
is distributed as Student's t-statistic with 1-2 degrees of freedom, 


Fh Pi : 
where Sys So» and are the sample variances of two part-tests 


cle 
and the covariance between them respectively. Kristof derived this 
formula by the maximum likelihood method under bivariate normal 

assumptions for the alpha coefficient, but the formula can also be 


used interchangeably for the reliability coefficient under the ETEM 


mode]. 


For the general case, J > 2, nothing is known yet, and 
at present the simulation method provides the only way to investigate 
the sampling distribution of reliability estimates. Because equation 
(2.17) does not involve the error variance parameter directly, it may 
be hoped that the distribution given by (2.17) is still valid or 
approximately true under the ETEM model if the normality assumptions 
are not violated. In other words, it is hoped that the distribution 
is robust against the violation of the homogeneity of error variances 


assumption to enable the test theorists to use the results obtained 


under the ANOVA model. 


To separate the effect of non-normality from that of non- 
homogeneous error variances, the ETEM model is first investigated using 
normal distributions of true and error scores. In order to make 
comparisons possible, the constants and parameters used for the cases 


of the ANOVA model are retained except for the values of the error 
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variances. With 3 levels of oa as under the ANOVA model, 6 different | 
sets of non-homogeneous error variances are used for the simulation 
experiments. The sets of error variances are given in the following 
table including the homogeneous case (EVI) used under the ANOVA model 


as a special case. 


TABLE 5.4 


Summary of Error Variances used Under ETEM Model 


Variance 
2 
(Nog, 


a” Kaye 
ex 


Error Variances foo 5} 


: 2 P p 
The last two columns of Table 5.4 give o> which is equal 


2 e e = 
to E(MS_), and the variance of fo.) within each set over J = 8. 
e 
To make comparisons easy, these sets are ordered with increasing degree 


of non-homogeneity, measured by the variance within each set, which has 
a range of 0.0 to 42.1875. 


Table 5.5 summarizes the mean and variance MS and 


MS for N= 2000 samples-in columns (1) to (4) inclusive, and 
e 9 


? 2 
compares the results with those obtainable from formula (5.6) with Oo, 
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TABLE 5.5 


Comparisons of Observed Means and Variances of MS's Under the ETEM Model 
and Normal Distributions With the Values Obtainable From 
Formula (5.6), N= 2000, | = 30, J= 8 
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replaced by rage and 1, = y= OF sine columns (6) andy (7) “Close 
agreement between expected MS's and the mean of observed MS's is seen 
as was the case for the ANOVA model. More specifically, the expected 

and observed variance of MS, columns (2) and (6), agree closely, but 
the observed variance of MS., column (4), differs greatly from the 
theoretical value obtainable from (5.6) given in column (7). The greater 
the non-homogeneity of error variances, the greater the discrepancy noted, 
reaching in the extreme a factor of three for experiment 21.  There- 
fore, it may be concluded that the formula (5.6) cannot be applied 


blindly in the case of the ETEM model, due to the possible effect of 


non-homogeneity of error variances. 


Table 5.6 summarizes the observed mean and standard error for 
each experiment in columns (3) and (4) and compares it with the values 
obtainable from (5.1), (5.3), and (5.10) given in columns (2), (5), and 
(6). It is observed that a rather close agreement exists between the 
observed mean of 6 and E(6) obtainable from (5.1)-(a) under the 
ANOVA model, i.e., columns (2) and (3), indicating robustness of the 
ETEM model as far as point estimation and biasedness are concerned. 

For the standard error of estimation, all two formulas predict the 
observed values reasonably well. Formula (5.10) seems better than 
(5.3), though the difference is not great. The calculated sum of 
squares of the deviation from the observed values are 0.00858 and 
0.00097. for formulas (5.3) and (5.10) respectively, confirming the 
conclusion. All of these results suggest that the standard error of 


reliability estimate is robust against the violation of homogeneity 


of error variances. 
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TABLE 5.6 


Comparisons of Observed Means and Standard Errors of Reliability Estimates 
Under ETEM Model and Normal Distributions With the Values Obtainable 
From Formula (5.3), and (5.10), N= 2000, | = 30, J=8 


Error Observed 6 S.E. by formulas p 
Set Mean SE. (5.3) (5.10) Se ae 
(4) (6) 

01 EV) 0. 0; OF 0. 0. 2 
02 | EV2 0. 0. 0. 0. 0. = 4.0 
03 | EV3 0. 0% 0: 0. 0. 
04 | EV4 0. OF 0. 0. 0. 
Oo. 1LEV5 0: 0. 0. 0. 0. 
06 | EV6 0. 0. 0: 07 0. 
07 | EV7 0. 0. 0. 0. 0. 
08 | Evi 0. 0. os 0. 0. 9 
09 | EV2 0. 0. 0. 0. 0. = 1.0 
10 | EV3 0. 0% 0. 0. 0. 
| Ev4 0. OC. 0. 0. 0. 
Ta} EVS (a). 0. 0. 0. 0: 
13 | EV6 0. (O)F 0. 0. 0. 
14 | EV7 0. 0. 0. 0. 0. 
15 | EVI 0. 0. (De On 0. 9 
16 | EV2 0. 0. 0. 0. 0. Oc 0.36 
WP A Vays! 0. Oe 0. 0. 0. 
18 | Ev4 0. 0. 0. @), 0. 
19° | EV5 0. 0. 0. 0. 0. 
20 | EV6 0. Oo; 0. 0. 0. 
2) EV7 07 0. 0. (0) 0. 
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Table 5.7 summarizes the lower and upper 5% portions of the 
observed distribution of 6 in columns (2) and (3) and compares them 
with the values obtainable under the ANOVA model and normal theory given 
in columns (4) and (5), namely from formula (2.17). The table also 
gives approximate real Type one error in columns (6) and (7) when the 
F-test of (2.17) is used for the ETEM model with normal distributions. 
The results clearly indicate the robustness of the F-test against the 
violation of homogeneity of error variance assumptions. Although there 
is a case (experiment 4) which gives as much as an 8% level of Type 
one error, there seems to be no systematic inflation or deflation of 


the nominal Type one error as a whole. 


Se 2.5.7 Ettects. of. Non-Normality on ETEM Model 


In the previous section, it was seen that the effect of non- 
homogeneous error variances on sampling distribution of reliability 
estimates is minimal, and it was also seen in Section 5.1 that the sampl- 
ing distribution is sensitive only to the violation of the assumption 
of the normality of true scores and is robust against distributional 
assumption of error scores. Therefore, it is logical to expect that 
the distribution is not robust against the distributional assumption 
of true scores, but the effect of non-normality of error scores must 
still be investigated under the ETEM model, since there is a possibility 
of interaction between the non-normal error score distribution and non- 


homogeneous error variances. 


To investigate this interaction effect, further experiments 
were carried out using the EV2 error variances set, chosen because 


2 
its oa. = 4.0018 is closest to o. = 4.0 used for the ANOVA model 


% iJ , Dy : 2 { } | : n ” : AN ; : 7 
\ : my , : | , . j : ' ‘9 t 
ee ; . = 7 See | 
f gis m J ; | i 
i al . i > i ‘at 

i ¥ : i —_ j i 

{ i 

‘ 


Ne edt to anotsya oh vaggy bee rewol oie zoe ivenmia ie -h oldeT 
gael esreqinod ‘brie. Me), ‘as. (s)! eam 1, ea noiawdinae7b 6 
nowin: «ett Henne bine fsbo avons ait spbew pidgnigsdo zauilew ok 

bale atdes ‘ont Be gy shone} mo ‘fen 42) bane @ e 
ada sew AM), brs, (8). anmutoa ni, TONS, ano. eayt Isen orsmixoraa # 


Janotsualsizib fein ion dain ‘Tsbom: Mata eas ot bees ai (ts) 9 
t ond jenkeps sesH?4 a to eesnaeudon, ait steaibat dienes ast 
svors- depart A: anoi:sqnuees soa6i7ev, Alors to vi Tanagonort tot 

eqyT 39, Lane $8 16 26 foun on zeviv AoTHy. a Fadil aad “a 
bt) ndiisftsb. 4o. nol 8) tri stadineltbys on ad O2 emeae snot. 4 VOT 


aE & 26 10419 bne aay 


Jebou MTS no! Jibteaieal 


| | a) 
-non te so0tte ads tert nee 2ew ti. ‘Wiofaane eusivera “ ate r cm 
: 


» ytd i tdatted to noldudiaaetb ani Teme fo aang ysy ron 
| sTemnee ads gehs 0.2 notasse ni naee als: z5w- uf bre _ Osanna ai 2s 
not tqnuees ait to not raloty-ads os yin ovistenee ‘et noltudtaseib g 


ae 


. (snot sudl verb denispe teudo 21 bis esn098 vial * ys emsan (3:3 


le j 


‘tert apogee 01 heaigor. zi ai setoveyedT -20ieba hina to mois 
“nolsgnuzes leagiaua seth ods daniees jeudor Jor ai ning 
teem eai00e IOI te (Metinon-nei to j287% ods Jud webrose ahs 
“yet itegoa & et ésieten esnia’, bsbom mata ens ebaw besspiseavel od vis 
non bee notauaivieib enooe rot9 teman-non an naowied noi fooresat 


a: | | 29061 76v 10719 


ae te ie y 
7 ! ‘atte nasil i987? noisser9I ni aida aregtdanve! ot 


‘aay ak iedes & y= 8> 03 “daszot ai “00.8 ap i 


_ ia oe 


TAREE *5'37 


Comparisons of Observed Lower and Upper 5% Critical Points and Real Type One 
Errors of F-Test When Nominal Value is Fixed at 5%, Under ETEM Model and 
Normal Distributions With the Values Obtainable Under ANOVA Model, 

N= 2000, | = 30, J=8 


Observed c.P.! Theoretical Cre Real Sig. (%) 
Lower Upper Lower Upper Lower Upper 


(2) (7) 
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2theoretical lower and upper 5% critical points of 6 under ANOVA model. 
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to make comparisons simpler, and three levels of oF, 


of true and error score distributions, namely uniform (U1), normal (NO), 


for three types 


and exponential (EX). Altogether the results of 27 experiments are 
summarized by tabulating the MS's (Table 5.8), standard errors (Table 5.9), 
and lower and upper 5% critical points of the distribution of reliability 
estimates with approximate real Type one errors when the nominal values 
are fixed at 5% level (Table 5.10). 

These 27 experiments may be compared with the results of the 
corresponding experiments under the ANOVA model, namely experiments 1-3, 
13-21, 31-39, and 49-54 of Tables 5.1, 5.2, and 5.3. The expected 
values of MS's and variance of MS) show close agreement with observed 
values, but formula (5.6) does consistently underestimate the variance 
of MSo; though the difference is trivial. Table 5.9 suggests that 
formula (5.10) closely approximates the observed standard error as in 
the case of ANOVA model. Observation of Table 5.10 also suggests that 
the pattern of discrepancy of real Type one error from the nominal 
value of 5% is almost the same as for the case of the ANOVA model, thus 
indicating non-existence of interaction effects between the non- 


homogeneous variance and non-normality of error score distributions. 


5.2.4 Conclusions for the Distributions Under ETEM Model 


The effects of non-homogeneous error variances on the 
sampling distribution of reliability estimates was investigated by 
simulating 21 experiments using three levels of p and 7 sets of 
error variances whose variance ranged from 0.0 to 42.1876. The 


following conclusions are tentatively made. 
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TABLE 5.8 


Comparisons of Observed Means and Variances of MS's Under ETEM 
Model with EV2 Error Variances Set and Various Combinations 
of True and Error Score Distributions with the Values 
Obtainable from Formula (5.6), 
N= 2000, | = 30, J = 8 


Parameters, 

Expected Values 

Under ANOVA 
(7) 


2 
c= 4.0 


po = 0.8888 
E(MS ,) = 36.0 


E (MS) = 4.0018 
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TABLE 5.9 


Comparisons of Observed Means and Standard Errors of Reliability Estimates 
Under ETEM Model With EV2 Error Variances Set and Various Combinations 
of True and Error Score Distributions With the Values Obtainable From 
Formulas (5.3), (5.1), and (5.10), N= 2000, | = 30, J=8 


Calculated from formulas 


(5.3) (5.1) (5.10) 


Parameters 


(3) (4) (5) 

0. 0.038 0.036 0.030 - 
0. 0. 0.038 0.036 0.030 
0. 0. 0.038 0.036 0.030 
0. @° 0.038 0.036 0.036 
0. 0. 0.038 0.036 0.036 
0. 0. 0.038 0.036 0.036 
0. 0. 0.038 0.036 0.057 
0. 0. 0.038 0.036 0.057 
0. 0. 0.038 0.036 0.057 
0. 0. 0.101 0.108 0.098 
0. 0. 0.101 0.108 0.098 1.0 
0. 0. 0.101 0.108 0.100 
0. 0. 0.101 0.108 0.108 0 = 0.6666 
0. 0. 0.101 0.108 0.109 E(6) = 0.6419 
0. 0. 0.101 0.108 0.109 | 
0. 0. 0.101 0.108 0.146 
0. 0. 0.101 0.108 0.147 
0. 0. 0.101 0.108 0.148 
0. 0. 0.151 0.188 0.180 
0. 0. 0.151 0.188 0.182 
0. 0. 0.151 0.188 0.189 
0. 0. 0.151 0.188 0.187 
0. On 0.151 0.188 0.188 E(6) = 0.3754 
0. 0. 0.151 0.188 0.195 
0. O. 0.15) 0.188 0.216 
0. 0. 0.151 0.188 O20? 
0. 0. 0.151 0.188 0.224 
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TABLE 5.10 


Comparisons of Observed Lower and Upper Critical Points of Reliability Estimates 
and Real Type One Errors of F-Test When Nominal Value is 5%, Under ETEM Model 
With EV2 Error Variances Set and Various Combinations of True and Error 
Score Distributions With the Values Obtainable Under the ANOVA Model 
and Normal Theory, N = 2000, | = 30, J = 8 


| 2 
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(a) The variance of MS. is sensitive to the violation of the 
homogeneity assumptions, and formula (5.6) should not be used to 
calculate this statistic. 

(b) For the point estimation of reliability, the ANOVA model is 
quite robust against the violation of homogeneity of error variances 
provided that the distributions are normal. 

(c) The standard error of estimation is quite robust against the 
violation of the homogeneity of error variances. The best formula is 
Seilt (5.10). 

(d) Formula (2.17) can be used freely without inflating or 
deflating Type one errors too much for the ETEM model provided that 


normality is not violated. 


The effect of non-normal true or error score distributions 
under the ETEM model was investigated by performing 27 experiments with 
three levels of op, three types of true and error score distributions, 
and a set of non-homogeneous error variances, The following 
conclusions are tentatively made. 

(e) Formula (5.6) consistently underestimates the variance of 
MS. under the ETEM model. 

(f) The interaction between the ETEM model and non-normal error 
score distribution seems negligible. 

(g) The conclusions drawn in Section 5.1.0 may be generalized to 


the ETEM model with little modification. 


5.3.0 Relaxation of the Homogeneity of True Variance Constraint in 


the ANOVA or ETEM Models 
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ee aa Reliability and the Alpha Coefficient 


In Chapter Two, the ANOVA and ETEM models were expanded to 
include more general models such as the congeneric or multi-factor true 
score models through the use of the vector or matrix parameters A and 
A in equation (2.6') to produce (2.28). Under these more general models, 
the ETEM assumptions are not satisfied in general, and the Alpha 
coefficient is lower than the reliability coefficient. Therefore, one 
might be interested in two related but different distributions, namely, 
the sampling distributions of the Alpha coefficient estimates and the 
reliability estimates. However, the Alpha coefficient has attracted 
test theorist's interest only because it is considered a practical, 
and easily computable substitute for the reliability coefficient. Thus, 
the distribution of the Alpha coefficient estimates is meaningful only 
in lieu of the distribution of reliability estimates. Furthermore, 
because no direct estimation formula for reliability is available under 
these more general models, without exception Alpha coefficient estimates 
have been accepted as reliability estimates regardless of the underlying 
models or assumptions. 

Test theorists know that the population Alpha coefficient 
is in general lower than the reliability, but this fact has been 
frequently confused with underestimation due to biasedness of the estimation 
procedure. Two kinds of underestimation problems that exist in reliability 
theory must be distinguished: one is due to deviation from the ETEM 
assumption, which is not a statistical inference problem, and the other 


is due to the nature of the estimation formula which is biased. 


The sampling distribution of the Alpha coefficient under the 
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non-ETEM model is the most overlooked aspect of reliability theory. 
No study has yet been reported on this subject to the author's know- 
ledge. Due to the mathematical complexity involved in these models, 
it seems almost impossible to investigate the problem by analytical 
means. Therefore, the problem was investigated as assumption violat- 
ing cases of the ANOVA model using computer simulation techniques. 
The major purpose is to find the effects of the violation of the ETEM 
assumptions, or homogeneity of true score variances and unifactorness 
of the true score dispersion matrix. 

Because so many assumptions of ANOVA models are violated 
under these more general models, an exhaustive investigation of al] 
Here obi nator of possible violation of assumptions is prohibitively 
expensive with the computer simulation method. The study in this 
section is limited to a few combinations. Therefore, the findings in 


this section have limited value for generalization. 


5.3.2 Distributions Under the Congeneric Model 


Under the congeneric true score model, each part-test measures 
the same trait except for the errors of measurement, i.e., the factorial 
structure of true scores is unifactor. Therefore all part-test scores 
have linearly related true scores. Test scores under the classically 
parallel, ANOVA (or essentially parallel), or ETEM models are all 
special cases of the congeneric model, as discussed more fully in 
Chapter Two. In these special cases any true score of a part-test must 
be essentially identical for a given subject, unlike the congeneric model. 

Zz 2 


Under the congeneric model, the variance, oN = ue On 


true score for part j depends on j, and there is not a common 
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2 2 ; 
variance parameter on which has played a key role in the ETEM or 
ANOVA models. To obtain the corresponding parameters for the congener ic 
Zu : 
model, a new parameter o, is defined denoting the average of the all 


elements of the dispersion matrix A A', namely, 


(5.13) HOC OUI Oe Gils Dae IY Cole Daye 
j ie J J 


As this parameter is an average of true score variance and covariances 


> 


the reliability coefficient is, 


POS UN Ad J on 
5.14) eo eS Sa : 
QA + ail Si on + J of 
where o is the average of error variances as defined by (2.22), 


namely the mean of (oo) . Since the distribution (2.17) obtained 
under the ANOVA model and normal distribution theory does not directly 
involve the parameters on and oF, but only directly involves the 
reliability oe, it is desirable to know whether the distribution 

of reliability estimates based on formula (2.13)-(b) is robust against 
the violation of ETEM assumptions, i.e., whether the relation (2417) 


still holds approximately for the congeneric cases. 


Under the congeneric model, formula (2.13)-(b) gives the 
estimate of the Alpha coefficient, not the reliability, but it is 
hoped that, with moderate violation of ETEM assumptions, inferences 
based on the estimate of Alpha would not invalidate the inferences 


of reliability too much as in the case of the previous section. 


To see the effects of non-homogeneous true score variances, 
sampling experiments were performed using the following three sets of 


A's representing three levels of reliability, namely, 
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which gives three levels of of namely 4.0, 1.0, and 0.36 and three 


levels of p, i.e., 0.8889, 0.6667, and 0.4186 respectively. The A's 


were chosen such that the values of fe equal of used for the ANOVA 
and ETEM model experiments in Sections 5.1.0 and 5.2.0, in order to 
facilitate the comparisons. The error variances (05, are fixed at 
4.0 as the ANOVA model, and the same constants are used for N, |, and 
Jf Mics, 2000, 30, and 8 respectively. Employing three types of true 
and error score distributions, namely uniform (UI), normal (NO), and 
exponential (EX), altogether 27 experiments were performed by RELOI, 

and the results are summarized in Tables 5.11, 5.12, and 5.13. As 

in the previous sections, the distributions of MS's are examined 
first. From Table 5.11, it is noted that the effects of non-homogeneous 
true score variances are minimal, i.e., the results are almost identical 
with those under ANOVA model given in Table 5.1. Table 5.12 summarizes 
the means and standard errors of reliability estimates under this model 
and compares them with the values obtainable from formulas (5.3), (5.1)-(b), 
and (5.10). It is clearly noticed that formula (5.10) is still the best 
among the three. When the means of 6 in Table 5.12 are compared 

with the corresponding values of Table 5.2, it may be noticed that 

under the congeneric model the mean of 0 is lower than under the 

ANOVA model, as expected, since the formula used for the estimation, 


(2.13), is for the estimation of Alpha, and Alpha is lower than 
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TABLE 5.11 


Comparisons of Observed Means and Variances of MS's Under the Congeneric Model 
and Various Combinations of True and Error Score Distributions With the 
Values Obtainable From Formula (5.6), N = 2000, | = 30, J= 8 


Observed MS Observed MS. i Parameters 
Mean : and €(MS) 
(3) 


p = 0.8889 
Alpha = 0.8870 
E(MS,) = 36.0 
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VABEEN 5S. 12 


Comparisons of Observed Means and Standard Errors of Reliability Estimates 
Under the Congeneric True Score Model With the Values Obtainable 
From Various Formulas, N= 2000, 1 = 30, J=8 


Observed 6 S.E. by formulas Parameters and expected 
(G3) (Saye) values under ANOVA 


(4) 


0 = 0.8889 
Alpha = 0.8870 
E(6) = 0.8807 
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reliability under the congeneric model. However, as can be seen in 

Figure 5.1, the shapes of the distributions are almost the same as 
expected from (2.17), namely under the ANOVA model and normal distribution 
theory. Therefore similar conclusions as cited in Section 5.1.5 may 

be obtained from the observation of Table 5.13, namely the real signif- 
icance levels of the F-test, or the lower and upper 5% critial points 


of 6 are almost the same as_ the values under the ANOVA model. 


To make the comparisons between the ANOVA model and the 
congeneric model, and to separate the effects of non-homogeneous true 
scores variances from the effects of non-homogeneous error variances, 
further experiments were performed under the same conditions as the 
ANOVA model cases except that the true score variances were allowed to 
differ. However, there is some possibility of the existence of inter- 
action effects between the effects of violating the two homogeneity 
assumptions, although each case was found to be quite robust against 


the violations. 


To investigate this problem, 15 additional experiments were 
performed employing three sets of A's as before and five sets of 
non-homogeneous error variances used in Section 5.2.0, namely EV3, 
EV4, EV5, EV6, and EV7. The results are summarized in Tables 5.14, 
5.15, and 5.16. When the entries of these tables are compared with the 
corresponding values of Tables 5.5, 5.6, and 5.7, little difference 
is noted between the two sets of values suggesting non-existence of 
such interaction effects. For example, experiment 3 of Table 5.5 gives 
the observed variance of MS, as 88.222, while the corresponding 
value under the congeneric model is given in experiment | of Table 5.14 


as 89.270. Therefore, it may be concluded that, the effect of non- 
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TABLE 5.13 


Comparisons of Observed Lower and Upper 5% Critical Points of Reliabillty Estimates 
Under the Congeneric True Score Model with the Values Obtainable Under the 
ANOVA and Normal Theory, and Real Type One Error of F-Test When the 

Nominal Value is 5%, N = 2000, | = 30, J= 8° 


TrueSige(%) Observed-C.P+!-|-Theoretteal- CePx- 
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2Theoretical lower and upper 5% critical points under ANOVA with normal distribution 
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TABLE 5.14 


Comparisons of Observed Means and Variances of MS's Under Congencric True Scores, 
Non-Homogeneous Error Variances and the Normal Distributions With the Values 
Obtainable Under ANOVA Model, N = 2000, | = 30, J=8 


Observed MS Observed MS, 


Parameters 


No. | Type | Mean Var. Mean Var. 
Le Oy nt) (8) 


01 | Ev3 36.10) 89.270 | 3.832 0.224 0. 
02 | EV4 35.611 83.103 | 3.630 0.239 0. 
03 | EVS5 36.308 87.363 | 4.321 0.44) 0. 
O4 | EV6 38.742 103.316 | 6.472 0.708 0. 
05 EV7 36.915 96.880 | 4.819 0.618 0. 
06 | EV3 11.795 9.654 | 3.758 0.24) 0. 
07 | Ev4 11.633 9.566 | 3.574 0.223 0. 
08 | EVS 12.249 10.12] | 4.693 0.432 0. 
09 | EV6 14.477 15.206 | 6.376 0.702 0. 
10 | EV7 12.740 VB32 | 42775 0.609 0. 
1) | EV3 6.700 3.203 | 3.768 07221) 0 
12 | EV4 6.430 2.779 | 3.564 0.232 0 
13. | EVS 7.120 3.49) | 4.258 0.430 0 
14 | Ev6 9.306 5.915 | 6.394 0.771 0 
15 | EV7 7.593 3.782 | 4.753 0.620 0) 

(MG Were G7) = 3.750 (ENS) 

oy ne 3.563 (EV4) 

4.250 (EVS) 

6.375 (EV6) 

4.750 (EV7) 


(a) var (HS,) = [27+ + fo7%q + (I-0)7¥_/S)] oq + 06)” 
(5.6) : 


meee vey 


(i-1¥Qel) 


(b) Var (MS) of 


at ry sugat 


ee sdostosd auitt 3 foesipniod sein 22M To operates $s! een schleaitie ; 
eautev) ade. HW endttudings sit! Van iat “grt tore @eanelae eid ate: B 
Ba (OR = 1,- ,QU0S eM shalt AVON ara aa wi - 


ayaa 


fa Pa sm pe 


acy 2) Wd a 6V 


en bavysedd 4 
ea eae 


ee 


aaa | a \ 
a, nati, | 


fa @) | Ola , 


"BEB SM, a ae | etd | SEB, fF OVeJOR = PORE 
: eh. ONE. aie: RES DESI CE | e0t. ad thee 
Bri.9 © asda a Oaa,08) (eo RECA eOeee 7 
Hosa | ike. re RENE] BOG at cb ae ity 
SSiid ht .ee |) OSt DE) Bie. | ‘088i aj 8 OE 
|, BHO S82. O2t(T hh RO. aOR | 
‘SSE,O © OSs.e Soe; (1h  ESRle i 2 ‘esa 
BXE.O. PAELOt | Das. st seh ap a = 
wo, fee. eyeval Pa js a ans, “t cn 


te ee Phas Ph re 0 


nd 


| ha * coulteutyys 3" tet) tay te ay “ ne v8 te) a 


b . eoRie 4 ie ara 
7 ep heen M (24) xa ba 
i | , 
Vi 
Poe a ee : | 
OF 2 : . 


106. 


TABLE 5.15 


Compartsons of the Observed Means and Standard Error of Rellability Estimates 
Under Congeneric True Score, Non-Homogencous Error Variances and Normal 
Distributions With the Values Obtainable From Formulas (5.1), (5.3), 


and (5.10), Ne& 2000) lee 30,0 = 6 
ai 798 ae Observed 6 SolBo. 1h7 
(4) (5) (6) (7) (8) 
0.893 0.887 0.886 0.036 0. 
0.898 0.892 0.89) 0.034 0. 
0.881 0.874 0.873 0.041 0. 
0.832 0.822 0.821 0.055 OF 
0.869 0.861 0.860 0.048 0. 
0.679 0.657 0.658 0.108 0. 
0.690 0.669 0.670 0.102 0. 
0.652 0.627 0.628 0.114 OQ. 
OF555 0.524 0.528 0.140 OF 
0.626 0.600 0.601 0.119 0. 
0.434 05393 0.397 0.180 OF 
0.446 0.406 0.409 On173 0. 
0.403 0.360 0.365 0.181 0. 
Ons 0.260 0.268 0.207 QO. 
(3/7) 0.331 0.337 0.190 0. 
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TABLE 5.16 
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Comparisons of Observed Lower and Upper 5% Critical Points Under Congeneric True 
Scores, Non-Homogeneous Error Score Variance, and Normal Distributions With 
the Values Obtainable Under the ANOVA and Normal Theory, and Real Type 
One Errors of F-Test When the Nominal Value is 5%, 


N = 2000, 
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homogeneous true score variance, or violation of ETEM assumption wil] 
affect the sampling distribution of the reliability estimates given by 
(2.17) very little as long as the degree of non-homogeneity is within 


a moderate range as with the A's used in these experiments. 


5.3.3 Distributions Under the Multi-Factor Model 


Classically, the assumption of a one-factor true score has 
been referred to as one which produces 'unit rank correlation matrix' 
(e.g., Kuder and Richardson, 1937), but as seen in Chapter Two, the 
unifactorness of true scores is inherent to the ANOVA linear model and 
its more general form such as the ETEM or congeneric models. Under 
these models, it is implicitly assumed that the test measures only one 
trait, and therefore, the true score can have only one factor structure. 
However, in real test score data, it is seldom possible to separate 
measurement of one trait from others. The psychological or achievement 
tests usually measure more than one trait at a time, and it is sometimes 
unrealistic to assume that only one factor exists and to regard all 
other factors as error. This fact has been well demonstrated by the 
rejection by many researchers of Spearman's so-called g-factor theory 
in modern factor analysis. Thus violation of unifactor true score 
assumption may not be considered simply as an exceptional case; this 


may be rather a common case for real data. 


In Chapter Two, the multi-factor test model has been introduced 
as a generalization of the congeneric test model by expanding the 
linear model of ANOVA step by step to a factor analytic model. However, 
since most of the test theories are based on the unifactor true score 


assumption, no reliability theory has ever been developed under this 
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model. Therefore the multi-factor model has been referred to as an 
assumption violating case of the classical model rather than a separate 
model in its own right. Following this traditional lines in tthisesitudy, 
the reliability distribution under the multi-factor model is treated 

as an assumption violating case of the ANOVA model as are other models 


examined in the previous sections. 


Since the Alpha coefficient is a measure of the first factor 
concentration (Cronbach, 1951), the coefficient is expected to be much 
lower than the reliability coefficient if second or higher factor is 
not negligible. Therefore, it is hardly expected that the sampling 
distribution of Alpha coefficient, as a substitute for the reliability 


estimate, is robust against the violation of the unifactor assumption. 


To support this conjecture, a number of sampling experiments 
were performed and the results are compared with those obtainable under 
ANOVA model. The parameters on and o are not defined under this 
model as with the congeneric model case, but the average of true and 
error score variance may be used to determine the effectiveness of the 


ANOVA model under the multi-factor model, namely, 


2 2 
a8 ' I = : J Y 
(Sens) PRG a des at) ( L 7 1 ve Mei d/ 


and oe as in the previous section. 
e. 


: 2 Z 
lf these parameters are used in place of On and cae most 


of the formulas introduced in Section 5.1.0 can be used directly and 
the robustness of the ANOVA model under multi-factor true score cases 


can be examined empirically by the simulation techniques. 
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Using the following two A matrices, an error score standard 
deviation matrix ¥Y,, and three types of true and error score distributions, 


altogether 18 experiments were performed under the multi-factor true 


score model with N = 2000, | = 30, J = 6, 
OF687 | 08302 0.4435 0.1510 
0.410 0.663 0.2050 0.3315 
feo 0.2421 0.4735 osha | pp cane O- 1270 0.3675 
—| | 0.369 0.816} ° —2 2-I |0.1845 0.4080 
0.417 0.557 0.2085 0.2785 
0.669 0.482 0.3345 0.2410 
0.34942 0.0 0.0 60 OO 0.0 
0.0 0.62636 0.0 0.0 0.0 0.0 
oa! |) Oa? 0.0 0.63341 0.0 0.0 0.0 
— | 0.0 0.0 0.0 0.44495 0.0 0.0 
0.0 0.0 0.0 Ole 0.71823) 0.0 
0.0 0.0 0.0 0.0 0.0 0.56579 


Table 5.17 compares the observed means and variances of the MS's under 
the multi-factor model with the values obtainable from formula (5.6) 
treating the model as an ANOVA model. It is noted that rather close 


agreement exists between the means of the observed MS and E(MS,) 


A 
given in colums (1) and (7), but the agreement is rather poor between 

the means of observed MS, and E(MS_) given in columns (3) and (7) 
indicating the effect of the violation of unifactor true score assumption. 
Even for the normal true and error score distributions, the difference 
between the two values are too big to be explained as sampling fluctu- 
ation. For example, experiment 5 gives the mean of MS. as 0.413 

while the theoretical value of E(MS.) = 0.3249 if the ANOVA model 

and he are used. This implies that the E (MS) undervalues the 


real expected value of MS: When the variances of MS's, in colums (2) 


and (4), are compared with the values obtainable from formula (5.6) 


inte ar022 | 079 6. <esvivven owt eatwol iat eda, ene f 
noinusiele erent yond bre. uid to aoqy? - aos bre cyt xtyem 
oe soagetsa on orig siphos bemi6?18q atew etnem! y9qxs a 

| ak (a= 1 40008 =H dni 


‘Jorat.o agaa.o}. 
‘1a(ge.0  Oa0s.0 
2X8E.0 9 OFS1.0) , 
‘J aars.o a8. | 


Oe 8.9 6.0 0.0: ila 
Pee Cs es aed 0.0 0.0 ey 
0.0 0.0 0,0 . EEE. 0 sl 
| i, a 0.0 Aevate | 0. 0,0 = 
bh ee Es8rt. ol 0.0 * 
imal st aR A 0:0 


1 


‘. ay etumio? aan Ne acta oul ey sit asiw {ebom RE: 


Adi 
Wy eae. iP 


“ peols erls61 ten3 heson et v1 datas AvOMA ne 26 tabom on ih 69% 


(ena bas. 2M ‘bovisedo ‘st to ensan sd ‘iii eheixe' ee 


A 


| * 
re nisswied 009, jodi8y ai dnamasnes. ott gud. teh brie (1) pay 7 44 

| iw rs 

a bins. @) enemy 103 ai nevi \, ana bie an bevisedo 0 ensem | 


“not sqnuees 7092 ound nodab ny 1) foie loty ad +0  abetie and, ents 


a catahies ott “anol sud er age worn ‘ies aut Fainvon odd ai 19V 


a uaaut? enitgnse aa bonietgxa ad os pid oor sis eowliev wd orld | ii wied 
; ie ian 
i? 0 oie" 


oe eee , eh to neom is 2evilp 2 Inet taqKe ‘shamed: 107 
a Jona edd ¥ eoseo (203%. obi 6y Ispisoroeds Any | 
+ (e3., bila sedi asi i qmi ait ‘ybgeu 28 . mth 
= » seane) ney tt nth ugh, 26 outed bedoeq us 4 
tort stdeniasdo seutev asd tw: asi ot a) by 


= 


cod 7 i r f j 1, id . 7 ‘ 
cia et. lee wt oon y 
co le y 7 7 - of .) Ne re 7 


1a. 


TABLE 5.17 


Comparisons of Observed Means and Variances of MS's Under the Multi-Factor 
True Score Model and Various Combinations of True and Error Score 
Distributions With the Values Obtainable Under ANOVA Model by 


Formula (5.6), N= 2000, | = 30, J=6 
ERB. Dis Observed MS, Observed MS. Var. by (5.6) | Parameters 
No. Tr. Er: Mean Var. Mean Var. MS MS and (MS) 
(1) (2) (3) (4) (5) (6) (7) 
3. 0 0.436 0.0017} 0.542 0.0008 on = 0.6001 
34 0 0.414 0.0026] 0.544 0:0015 2 
3. 0 0.414 0.0067] 0.548 0.0050 of” = 0.3249 
2 | 0.413 0.0019 | 1.062 0.0008 a 
Shs ] O413 ~ 0.0027 | 1-063 0.0015 E(MS,) = 3.9235 
an | 0.413 0.0066 | 1.066 0.0050 
Bt 2 Och Vive § 0.10027 te3 2655 0.0008 0.3249 
3), 2 0.414 0.0036 | 3.656 0.0015 
at 2 0.413 0.0073 | 3.659 0.0050 
1 0. 0.345 0.0010 | 0.070 0.0008 0.1500 
1 0. 0.346 0.0019 | 0.071 0.0015 : 
1 0. 0.346 0.0058 | 0.075 0.0050 0.3249. 
| 0. 0.348 0.0010 | 0.103 0.0008 ; 
1 0. 0.346 0.0019 | 0.103 0.0015 E(MS,) = 1.225 
| 0. 0.348 0.0059 | 0.107 0.0050 
1 0. 0.346 0.0011 | 0.265 0.0008 E(MS.) = 0.3249 
1 0. 0.347. 0.0020 | 0.266 0.0015 
] 0. 0.346 0.0063 | 0.269 0.0050 
2 2 DG: 
(a) var (MS) = [ppt 7 {0% + (I-p)” yQ/J}] oq + 96) 
(5.6) 
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given in columns (5) and (6), rather poor agreement is noticed, suggest- 


ing inapplicability of the formula. 


Table 5.18 gives means and standard errors of reliability 
estimates and compares them with the values obtainable from formulas 
(5.3), (5.1)-(b), and (5.10). It is observed that, for all experiments, 
the mean of 6 is much lower then population Alpha or E(6) under 
ANOVA and normal theory, indicating the effect of the multi-factor 
true score structure. This result is probably due to the fact that 
the Alpha coefficient measures mainly the variance due to the first 
factor, and thus underestimates the true score variance and over- 
estimates the error score variance, and at the same time shifting the 
distribution of reliability estimates considerably to the left as shown 
in Figure 5.2. Although formula (5.10) seems still to be the best 
among the three, the fit is very poor suggesting inapplicability of 
most of the formulas derived under the ANOVA model and normal theory 


for multi-factor true score test. 


Discrepancies between observed and theoretical distributions 
based on ANOVA model are clearly seen when the real significance level 
of F-test is compared with the nominal value of 5%, as summarized in 
Table 5.19. The real significance level for the lower tail range from 
L4.402: to 27.50% for Ay and 5.80% to 12.30% for os clearly 
indicating the inapplicability of the conventional F-test to multi- 
factor tests. For the upper tail, the true significance levels are 
in general lower than the nominal value, but the results are not 


predictable. For example, experiment 2 gives a value as low as 0.45%, 


while experiment 18 gives one as high as 8.10%. All of these results 
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TABLE 5.18 


Comparisons of Observed Means and Standard Errors of Reliability Estimates 
Under the Multi-Factor True Score Model and Various Combinations of 
True and Error Score Distributions With the Values Obtainable From 
Formulas (5.3), (5.1)-(b) and (5.10), N-= 2000, | = 30, J=6 


Observed 6 Calculated S.E. by Parameters and E(6), 
Mean CER (5.3) (5.1)-(b) = (5.10) }| ANOVA and Normal 
(oe (4) (5) (6) 


0.6001 
0.3249 
0.9172 


ah ie 
Alpha = 0.8943 
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TABLE 5.19 


Comparisons of Observed Lower and Upper 5% Critical Points of Reliability Estimates 
and Real Type One Errors of F-Test When the Nominal Value is 5% Under the 
Multi-Factor True Score Model and Various Combinations of True and Error 
Score Distributions With the Values Obtainable Under the ANOVA Mode] 
. and Normal Theory, N = 2000, | = 30, J =6 


| 


Real Sig. (%)] Observed C.P. | Theoretical ate 


Parameters 


No. tems attr Lower Upper Lower Upper Lower Upper 


Q) (2) (3) (4) (5) (6) 


0. 0.813 0.929 0 0. 
0. 0.828 0.930 0 0. 
Je 0.817 0.934 0 0. 
0. 0.820 0.931 0 0. 
QO. 02815. 0.932 0 0. 
ihe 0.819 0.936 0 0. 
ile 0.814 0.939 0 0. 
lle 057972 2 01.936 0 0. 
2. 0.786 0.939 0 0. 
foo 0.541 0.813 0 0. 
Jaf F530, 0.815 0 0. 
4, 0.5225) 0.2825 0 0. 
Ze 0.524 0.814 0 0. 
72. 0.508 0.814 0 0. 
h, 0.500 0.826 0 0. 
By 0.475 0.828 0 0. 
h 0.450 0.828 0 0. 
8. 0.466 0.845 0 0. 


lobserved lower and upper 5% critical points of 6. j 


theoretical lower and upper 5% critical points of 6 under the ANOVA model with normal 
distribution of true and error scores. 
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strongly suggest that ANOVA model and normal theory are not robust 


against the violation of the assumption of unifactor true score. 


5.3.4 Conclusions for the Effects of Non-ETEM Model 


Based on the above discussions, the following conclusions are 
tentatively made. 

(a) Formula (5.6) may be used for the congeneric test case if 
on and of are used in place of of and of of ANOVA model. How- 
ever, this formula is valueless for the case of multi-factor true score 
model. 

(b) The non-homogeneity of true score variance has little effect 
on the distribution, although the ETEM assumption is violated if the 
violation is moderate. The conclusions obtained in Section 5.1.5 may 
be generalized to the congeneric true score cases with moderate violation 
of ETEM assumption. 

(c) The effects of violation of the unifactor true score assumption 
are the most critical. If this assumption is violated, the formulas 
derived under the ANOVA model cannot be applied directly even with a 
normal true score distribution. 

(d) The F-test based on (2.17) may be used for the congeneric 
model if the true score distribution is approximately normal as in 
the ANOVA model case and the homogeneity of true score variance is 
satisfied approximately, but it would be misleading in multi-factor 
model cases. This is especially true for inferences based on lower 


portions or high reliability case. As previous sections showed, these 


effects diminish with the lower values of reliability. 


Findings in this section are based on rather limited 


combinations of possible parameters and distributions of true and 
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error scores, and therefore, generalization must be made with care. 


5.4.0 Summary 


Sampling distributions of reliability estimates for the 
continuous part-test cases are investigated under the ANOVA, ETEM, and 
congeneric and multi-factor true score models with various combinations 
of true and error scores distributions by analytical and computer 
simulation methods. Tukey's results for the calculation of the 
variance of variance estimate under an ANOVA model were applied to 
test theory to obtain an approximate formula for standard error of 
reliability estimates when the distributions of true and error scores 


are not necessary normal. 


To investigate sampling distributions of reliability estimates 
based on formula (2.13) under these models and distributional assumptions 
not necessarily normal, to see robustness of the ANOVA model and normal 
theory represented by the formula (2.17), and to evaluate the new 
formula for the standard error of reliability estimates, altogether 156 
experiments were performed by RELO], each requiring approximately 6 
minutes of computer C.P.U. time. From the experiments, the following 
conclusions may be obtained. 

(a) The equation (2.17) obtained under the ANOVA model and 
normal theory is quite robust against the violation of the following 
assumptions if the reliability estimate is based on (2.13), i.e., the 


estimation formula for Alpha coefficient: 


i) Normality of error score distributions. 
ii) Homogeneity of error score variances. 


iii) Homogeneity of true score variances, if violation is 
moderate. 
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But the ANOVA model and normal theory is not robust against violation 
of the following assumptions. 


i) Unifactorness of true score dispersion matrix. 


ii) Normality of true score distributions. 
The effects of the violation of these last two assumptions wil] 
decrease as the values of reliability decrease. 

(b) For the F-test based on the equation (2.17), the multi- 
factor true score model increases Type one error for the lower tail 
and decreases it for the upper tail by pairing the distributions of 
reliability estimates leftward substantially, when second or higher 
factors of the true score dispersion matrix cannot be ignored. 

(c) The effects of non-normal true score distributions depend 
on the magnitude of their kurtosis. For negative kurtosis, Type one 
errors for both tails are less than the nominal value, while for 
positive kurtosis, they are greater than the nominal value. The 
greater the absolute value of kurtosis, the greater is the discrepancy 
from the nominal value. 

(d) If true scores are distributed as normal, the ANOVA, ETEM, 
and congeneric models give almost identical distributions of reliability 
estimates with moderate departures from homogeneity assumptions of 
error and/or true score variances. 

(e) The new standard error formula (5.10) is superior to the 
traditional formulas (5.1) or (5.3), if the true scores are not 


distributed as normal. 
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CHAPTER SIX 


RESULTS FOR BINARY ITEM TEST SCORE CASES 


This chapter presents the results of computer simulated 
experiments for the binary item test score cases. Section 6.1 deals 
with the overall factors which might affect the distribution of reli- 
ability estimates. In Section 6.2, the effects of non-normal error 
distributions are investigated with normal latent score distributions 
and homogeneous biserial correlations. Section 6.3 deals with the 
cases Of non-normal latent scores with homogeneous biserial correlations 
and normal error scores, while Section 6.4 deals with non-normal latent 
scores and non-homogeneous biserial correlations. For all cases, both 
homogeneous and non-homogeneous item difficulty parameters are employed 


to determine the effects of non-homogeneous difficulty parameters. 


6.1 Factors Related to Binary Item Test Scores Distribution 


As discussed in Chapter Three, for a composite test con- 
sisting of J binary items as its part-tests, direct decomposition 
of observed score ine, which takes the value unity for a correct 
response and zero otherwise, into two independent parts, namely true 
and error scores, is impossible. Thus the linear model equation (3.3) 
can only be applied to an intervening variable or ‘response strength 
variable' Yai which is a hypothetical continuous variable. 

Under the normal ogive model, it was possible to evaluate 
test parameters such as the variance o, reliability p, and KR20 


by means of numerical methods if the item parameters, such as difficulty 
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parameters tn} and biserial correlations {rit, are specified. 
Unfortunately, however, the computational formulas given in Chapter 
Three are valid if and only if the normal ogive model is valid, namely, 
if the {f.} and fei) are independently and identically distributed 
as N(0,1) as discussed in Section 4.6 of Chapter Four. Thus the non- 
normal distributions of these two types of random variables would 
affect not only the sampling distribution of reliability estimates, but 


also the population test parameters. 


Furthermore, for the continuous part score case, the fixed 
constant for each part, oh indicates the relative difficulty level 
of each part-test, but these parameters do not enter any formula for 
reliability or any other test parameters, and are independent of the 
sampling distribution of reliability estimates. Therefore it was not 
necessary to consider the effects of {BJ on the distribution of 
reliability estimates. For the binary item case, however, the item 
difficulty parameters, the analogue of 4 for the continuous case, 
enter the formula (3.15) through threshold constants and consequently 
affect such test score parameters, as the mean, variance, reliability 
and KR20. Furthermore, as shown in Section 3.4 of Chapter Three, the 
ETEM assumption is satisfied if and only if the items are all homogeneous, 
namely they have equal difficulty and biserial correlation parameters. 
Therefore, if the difficulty parameters are not homogeneous, it is 
expected that the KR20 will be lower than the reliability and 
subsequently the sampling distribution of reliability estimates may 
differ from that of the homogeneous case, though there is some indication 


that the effects are not great (Nitko and Feldt, 1969). 
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As a result, for the binary item cases, the following factors 
must be taken into account for a study of sampling distributions of 
reliability estimates: 

(a) The effect of non-normal distributions of ce and: tec. 
|.e., the effect of the violation of the normal ogive model. 
(b) Homogeneity of item difficulty parameters and biserial 


correlations, i.e., the effect of the violation of the ETEM assumption. 


Obviously it is impossible to investigate the sampling distri- 
butions of reliability estimates under all possible combinations of the 
above factors and all possible sets of parameters by computer simulation 
techniques. In this chapter, to conserve the overall computer time, 
the experiments and investigations are limited to only three distribu- 
tions for {f.} and fe; ;}, namely uniform (U1), normal (NO), and 
exponential (EX); four sets of difficulty parameters, two of which are 
non-homogeneous, and six sets of biserial correlations, three sets of 
which are non-homogeneous. The parameter sets used for the experiments 


are given in Tables 6.1 and 6.2 


TABLE 6.1 


Item Difficulty Parameters 


Nota- | Homoge- Item Number 


tion | neity 4 5 


D1 Homo. 
Non-Homo. 
Homo. 


Non-Homo . 
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TABLE 6.2 


Item Biserial Correlations 


Homoge- ltem Number 
5 Mean Var. 


neity 4 


Homo. 
Non-Homo. 
Homo. 
Non-Homo. 
Homo. 


Non-Homo. 


With these distribution-parameter combinations, altogether 
216 experiments (3 x 3 x 4 x 6) are possible. However, previous results 
indicated that the error distribution has little effect on the distri- 
bution of reliability estimates, and the same tendency may be expected 
for the binary item cases. Since this was the case, as will be seen 
in the following section, only the first step of the investigation wil] 
involve the case of non-normal error distributions. Thus the total number 


of experiments run were 96, resulting in a saving of computer time. 


6.2 The Effects of Non-Normal Error Distribution and Non-Homogeneous 
Item Difficulty Parameters 
In order to separate possible effects of non-normal latent 
score distribution and non-homogeneous biserial correlations such as 
B2, B4, and B6, from those of non-homogeneous item difficulty parameters 
Or non-normal errors, which are of major interest in this section, three 
homogeneous biserial correlation sets Bl, B3, and B5, normal distribution 


of latent variables, four sets of difficulty parameters, three types 
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of error distributions are used, i.e., altogether 36 (3x4x3) experiments 


with N = 1000, | = 30 and J = 9 are performed. 


Table 6.3 presents population parameters calculated from the 
formulas given in Chapter Three and the results obtained from the 
parallel form method, with sample size 30030. Comparisons of data 
in Table 6.3 indicate: 

(a) Calculated test parameters based on formulas given in Chapter 
Three agree reasonably well with the results obtained by computer 
simulation, thus partially validating the computer simulation method. 
For example, experiment (2 was performed with normal error score 
distribution, and satisfies the normal ogive model... It gives the 
test score mean, variance, reliability, and KR20 as 4.491, 8.094, 0.813, 
and 0.812, while the theoretical values based on the normal ogive model 
are 4.5, 8.118, 0.813, and 0.813 respectively. 

(b) For normal latent score distributions, the observed test 
score means given in column (5) seem to depend only on the average of 
the item difficulty parameters as expected, and are affected neither 
by non-homogeneous difficulty parameters nor non-normal error score 
distributions. For example, the values of experiments  1- 6 inclusive 
in column (5) are almost identical to theoretical value 4.5, although 
experiments 1, 3, 4, ‘6 have non-normal error score distributions, 
and experiments 4, 5, and 6 have non-homogeneous difficulty para- 
meters. 

(c) The non-homogeneous difficulty parameter sets, D2 and D4, 
(e.g., experiments 4, 5, 6, and 10, II, 12) result in lower test 
score variance, reliability, and KR20 when compared with the same 


average level of difficulty, but homogeneous, namely DI, and D3 
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TABLE 6.3 


Comparisons of Calculated Test Parameters Under the Normal Ogive Mode] 
With Empirical Values Based on the Parallel Form Method, Normal 
Latent Scores, and Homogeneous Biserial Correlations, 
NI = 30030, J=9 


Observed by P.F.M. 


Mean Var. Rel. 
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3 
5 
5 
5 
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5 
5 
3 
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3 
3 
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respectively (e.g., experiments 1, 2, 3, and 7, 8, 9). For non- 
homogeneous difficulty, i.e., D2 and D4, the KR20 coefficients are 
lower than the reliability as expected since the ETEM assumption is 
not satisfied. For example, experiment 12, with non-homogeneous 
difficulty set of D4, gives reliability and KR20 as 0.787 and 0.780 
respectively. 

(d) For homogeneous item difficulty, the higher the item difficulty 
is above the ideal 0.5 level, the lower the test variance, reliability 
and KR20. The same trends are observed for difficulty lower than 0.5 
level, though the results are not reported in this paper since almost 
exact ly the same results as high difficulty cases are obtained for 
lower difficulty cases except for test means, i.e., the test parameters, 
except for the test means, are highest when the item difficulty para- 
meters are all equal to 0.5 which is a well-known fact in test theory. 
For example, experiment 1 has homogeneous difficulty of 0.5 for all 
items and gives variance and reliability as 8.044 and 0.811 respectively, 
while experiment 7, which is comparable to experiment 1 except the 
higher difficulty of 0.7, gives 6.493, and 0.802 respectively. However, 
this conclusion would not apply in general to the non-homogeneous item 
difficulty cases, i.e., the non-homogeneous item difficulty effects 
interact with the effects of item difficulty level, and the results are 
not predictable, as it can be seen when the results of experiments h, 

5, and 6 are compared with those of experiments 10, 11, and 12. 

(e) The non-normal distributions of error scores have very little 
effect on the test parameters. For example, experiment 12, which has 
an exponential error distribution, gives parameter values as 6.. 288 , 


5.689, 0.787, and 0.780 which can be compared reasonably well with 
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theoretical values given in columns (1)-(4) inclusive, namely 6.3, 
5.671, 0.788, and 0.780 respectively. Alternatively, they can also 

be compared reasonably well with the corresponding values of experiment 
11 which has a normal error distribution, namely 6.296, 5.632, 0.785, 


and 0.777. 


Table 6.4 gives the means and standard error of reliability 
estimates over N = 1000 trials and compares them with theoretical values 
which can be obtained from continuous part scores under the ANOVA model 
and normal distributional theory, i.e., treating binary test scores 
tx; 5) as if they were continuous part scores as in the previous 
chapter. From the table, it is noted that the observed means of 
reliability estimates given in column (2), which is based on estimation 
formula (2.13), compares fairly well with the theoretical values given 
in column (4) based on (5.1)-(a), the largest difference being only 
0.018 (experiment 32) which is probably too small to be meaningful in 
test theory. The standard error obtained from formula (5.3) or (5.1)-(b), 
given in columns (5) and (6), also predict the observed standard errors 
given in column (3) reasonably well, although formula (5.3) bao.” 
consistently underestimate the standard errors for lower reliability 
cases, namely the case of biserial correlation set B5. In general, 
formula (5.1)-(b) seems quite satisfactory, the largest difference 
between the theoretical and observed values being only 0.0111 (experi- 
ment 28). The sum of squares from the observed standard errors are 
0.00101 and 0.00956 for formulas (5.1)-(b) and (5.3) respectively, 
suggesting the superiority of formulas (5.1)-(b) to (5.3). No attempts 


are made to use formula (5.10) since neither kurtosis formulas of test 
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TABLE 6.4 


Comparisons of Observed Means and Standard Errors of Reliability Estimates 
Under Normal Latent Scores, Homogeneous Biserial Correlations With 
the Values Obtainable From ANOVA Model and Normal Theory, 
N= 1000, | = 30, J#9 


Bis Dif Rel .* Observed 6 E(6) by Expected S.E. by 
i ; Mean SzE. (5.1)-(a) (5.3) (5.1)-(b) 
Q) (2) (3) (4) (5) (6) 


0. 
0. 
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0. 
0. 
0. 
0. 
0. 
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0. 
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0. 
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0. 
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0. 
0. 
0. 
0. 
0. 
0. 
0. 
0. 
0. 
0. 
0. 
0. 
0. 
0. 
0. 
0. 
0. 
0. 
0. 
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*theoretical value if error scores are normal, otherwise the value was obtained by the parallel 
form method. 
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score for binary item test nor any numerical means to evaluate the 


parameter are available at present. 


Table 6.5 indicates the shapes of the lower and upper portions 
of the distributions of reliability estimates by giving the lower and 
upper 5% critical points of the distributions in columns (2) and (3), 
and by comparing them with those results obtainable theoretically from 
(2.17), given in columns (4) and (5). The results, in general, suggest 
that the theoretical values are very close to the observed values 
except for the upper tail portions for some experiments with high or 
medium reliability, i.e., with Bl and B2, and non-homogeneous difficulty 
set D2, namely experiments 4, 5, 6, 16, 17, and 18. For those 
experiments, the distributions are systematically shifted toward lower 
reliability primarily due to the fact that KR20 is substantially lower 
than reliability, because of extreme non-homogeneity of item difficulty 
parameter set D2. This is illustrated in Figure 6.1. Consequently the 
real Type one errors of the F-test for upper tails are much smaller 
than the nominal 5% level, some dropping as low as 1.1% level [column (7) 
of experiment 4]. The effect of non-homogeneous item difficulty 
parameters on the real significance level diminishes as the variation 
of item difficulty parameters decrease, as shown by experiments 10, 

11, 12, 22, 23, and 24. The effect of item difficulty also diminishes 
with lower reliability level, and no meaningful differences are observed 


for low reliability cases illustrated by experiments 25-36 inclusive. 


From the above observations, the following conclusions are 


tentatively made. 
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TABLE 6.5 


Comparisons of Observed Lower and Upper 5% Critical Points Under Normal 
Latent Scores and Homogeneous Biserial Correlations With the Values 
Obtainable From the ANOVA Model and Normal Theory, and Real 
Type One Error of F-Test When Nominal Value is Fixed to 
the 5% Level, N= 1000, | = 30, J =9 


Observed ee Theoretical c.P.2 


Lower Upper 


(2) 


Real Sig. (0/0) 
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‘Theoretical value if error scores are normal, otherwise the value was obtained by the 
parallel form method. 


~Observed lower and upper 5% critical points of the distribution of 6. 


Sthdoretticsl lower and upper 5% critical points of the distribution of 6 under ANOVA 
and normal theory. 
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FIGURE 6.1 DISTRIBUTION OF REL. EST.. NO. S OF TABLE 6.4% 
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(a) The effects of non-normal error distributions are small. 

(b) Formula (5.1), both for the expected value and standard 
error of 6, seems quite satisfactory for binary item cases although the 
assumption of continuity of observed scores is violated, provided that 
the latent scores are normally distributed, and the biserial correlations 
are homogeneous. 

(c) The item difficulty parameters tn} affect the distribution 
systematically, contrary to the previous findings reported by Nitko and 
Feldt (1969). In general, the heterogeneity of difficulty shifts the 
distributions to the left, and the more heterogeneous the difficulty 
parameters, the more distortion is observed, and it also appears to 
cause a large shift leftward for high reliability cases. If F-tests 
based on (2.17) are used with a fixed nominal significance level, the 
real Type one errors for the upper tail portion are affected by the item 
difficulty parameters. 

(d) The distributions of the lower tail portion for high or middle 
range reliability, or both tails for low reliability are quite stable 


against the heterogeneity of item difficulty parameters. 


6.3 Effects of Non-Normal Latent Scores 


The normal ogive model for the binary item test scores assumes 
the existence of latent variables or scores {f} distributed indepen- 
dently and identically as N(O,1). However, it is not conceivable that 
these assumptions are always satisfied. Therefore, the effects of non- 
normal latent distributions are one of the important factors which must 
be examined rather closely. For the continuous part score cases, it 
is known that the non-normal true scores affect the distribution of 


reliability estimates significantly, and inflate or deflate the real 
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Type one errors for the F-test. Thus it must be determined whether the 
same is true for the binary item test cases when the latent scores are 
not normal. Because it is known, from the experiments of the previous 
section, that the effects of non-normal errors are small, and to save 


computer time, experiments were performed using only normal error scores. 


Using two kinds of non-normal latent score distributions, 
namely uniform (U1) and exponential (EX), four types of item difficulty 
sets and three kinds of homogeneous biserial correlation sets were 
selected. A total of 24 (2x 4x 3) additional experiments were 
performed with N = 1000, | = 30, and J = 9. The results of these 
24 experiments are summarized in Tables 6.6, 6.7, and 6.8, together 
with the results of 12 experiments of the previous section which uses 


normal latent and error scores, for the purposes of comparisons. 


As in the previous section, the population parameters were 
first examined to determine the effects of non-normality of latent scores. 
From Table 6.6, it is clearly observed that the test means are almost 
identical for both methods, namely by theoretical calculations under 
the normal ogive model given in column (1) and by the parallel form 
method given in column (5), except for the exponential distributions 
which have non-zero skewness. Using the exponential distribution, 
the means are in general lower than the theoretical values suggesting 
the effects of skewness, since, unlike the variance, the means are in 


general more sensitive to non-zero skewness. 


The effects of non-normal latent scores can be seen rather 


clearly when the observed variance, reliability and KR20, given in 


columns (6), (7), and (8), are examined. The values of variance, 
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TABLE 6.6 


Compartsons of Calculated Test Parameters Under the Normal Ogive Model With 
Empirlcal Values Based on the Parallel Form Method, Normal Error Scores, 
Homogencous Biserial Correlations, NI = 30030, J = 9 


Theoretical (N.0.) Observed by P.F.M. 
Mean Var. Rel. KR20 Mean Var. Rel. 
(6) (7) 
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reliability and KR20 under uniform distributions are always higher than 
the corresponding values under normal distributions, while the values 
under exponential distributions are lower than the values under normal 
distributions. Thus the normal ogive model gives lower values for the 
uniform distribution cases than the real values and higher values for 
the exponential distribution. For example, with biserial correlation 
and difficulty parameters fixed to set Bl and D2, the theoretical values 
under the normal ogive model are 4.979, 0.769, and 0.752 for variance, 
reliability and KR20 respectively (experiment 5). The parallel form 
method under normal distribution gives 5.012, 0.772, and 0.754, closely 
approximating the theoretical values as expected. However, for uniform 
latent scores (experiment 4), the corresponding values are 5.342, 
0.788, and 0.772, which are much higher than theoretical [given in 
columns (2), (3), and (4) of experiment 4] or observed values under 
normal distribution of latent scores [given in columns (6), (7), and 

(8) of experiment 5]. On the other hand, for exponential distributions 
(experiment 6), the observed values, i.e., 4.247, 0.720, and 0.702, 

are much less than the theoretical or observed values under normal 


distribution. 


Therefore it may be concluded that the reliability para- 


meters to be used for equation (2.17) must be p*, the value obtained 


by the parallel form method, rather than p for non-normal latent score 


distribution cases, since these values are closer to the actual values 


than the theoretical values obtained under the normal distribution 
assumption of latent scores. 


Table 6.7 presents the results for observed means and stand- 


ard errors of reliability estimates using N = 1000, and compares them 
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TABLE 6.7 


Comparisons of Observed Means and Standard Errors of Reliability Estimates 
Under Normal Errors and Homogeneous Biserial Correlations With the 
Values Obtainable From ANOVA Model and Normal Theory, 

N= 1000, I! = 30, J=9Q 


Observed 6 E(6) by Expected S.E. by 
Mean S.E, (5.1)-(a) (5.3) (5.1) =(b) 


(3) (5) 
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with the calculated values based on formulas (5.1), and (5.3). It 
is noted that E(6) of (5.1)-(a) given in column (4) predicts very 
well the observed means of reliability estimates given in column (2) 
regardless of the distributions of latent scores, the largest dis- 
crepancy being only 0.019 (experiment 6), suggesting robustness of 
the estimation formula (2.13) as far as point estimations are concerned. 
The observed standard error of estimation given in column (3) suggests 
that the uniform distributions of latent scores produces smaller 
standard errors while the exponential gives larger standard errors 
than under the normal distributions for high reliability cases. 
Formula (5.3) or (5.1)-(b) predicts the standard errors of reliability 
estimates reasonably well, though (5.1)-(b) seems better than (5.3). 
Table 6.8 summarizes the shapes of the distributions of 6 
at the tall portions by comparing lower and upper 5% critical points 
given in columns (2) and (3) with theoretical values given in columns 
(4) and (5). From the table, it may be concluded that the effects of 
item difficulty parameters as noted in the previous section can be 
generalized to non-normal latent score cases. On the other hand, from 
the observations of the real Type one errors, the effect of non-normal 
latent score distributions are not so obvious. The Type one errors are 
fluctuating substantially, but with no clear sign of systematic inflation 
or deflation of Type one errors due to non-normal distribution of latent 
scores, unlike the case of continuous part scores discussed in Chapter 
Five. This suggests robustness of the ANOVA model and normal distribu- 
tional theory for the case of binary item tests. 


From the above observations, the following conclusions were 


tentatively made. 
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TABLE 6.8 


Comparisons of Observed Lower and Upper 5% Critical Points Under Normal 
Error Scores and Homogeneous Biserial Correlations With the Values 
Obtainable From ANOVA Model and Normal Theory, and Real Type 

One Error of F-Test When Nominal Value is Fixed to the 


5% Level, N= 1000, | = 30, J=9 

Observed C.P. Theoretical c.P.? Real Sig. (0/0) 

Lower Upper Lower Upper 

(6) (7) 

0. 0. oO; 3.80 6.30 
0. 0. 0. 3.80 7.40 
0. 0. 0. 4.60 6.70 
QO. 0. 0. 3.80 1.50 
0. 0. 0. 5.20 1.50 
0. OF 0. 50 3.20 
0. 0. 0. B20 6.70 
0. 0. 0. 6.70 6.80 
0. 0. 0. 3.70 5.70 
0. OF OF 4.00 2.90 
OF 0. 0. 6.40 4.90 
0. 0. 0. 31520 20 
0. OF 0. 3.00 Soll 
0. 0. 0. 3.90 5.60 
OF Plo 0. 5.40 5.50 
0. 0. 0. 3.60 190 
0. 0. Of 4.90 3.00 
0. 0. 0. 7.20 4.00 
0. OF 0. 3.80 5.60 
0. 0. 0. 6.20 7.90 
0. 0. 0. 4.20 3.80 
0. OF 0. 5.10 3.00 
0. 0. QO. 6.40 4.20 
(0) 0. 0. 2.90 3.60 
OF OF 0. 3.60 4.70 
0. 0. 0. 4.70 5.30 
0. 0. 0. 4.80 SLO 
0. 0. 0. 4.70 4.80 
0. Oy 0. 5.60 4.60 
0. 0. 0. 5.50 4.70 
0. 0. 0. 4.50 530 
0. 0. 0. 5.20 6.70 
0. = 0. 0. 3.90 4.90 
0. OF 0. 3.00 4.00 
0. 0. 0. 5.50 4,80 
0.363 |- 10: - 0. Se 4.00 


theoretical values if true scores are normal, otherwise the values obtained by the 
parallel form method. 


~Observed lower and upper 5% critical points of the distribution of 6. 


3théoretical lower and upper 5% critical points of the distribution of 6 under ANOVA 
and normal theory. 


avai 


Ismio isball pie tao istoo: 30 
zauleV on? Avi anotiahayiaa yt 1 a 
aqyt leah bas yTosnT Lema bie 
snd oa bsxit ef ea tection 
eat oe "= 


iA\ a] Neaiasicast 


j (o\o). ple (ken! 
neqqu: Seg bNOs 


4aaqu 1swol 


BNR IRS SIR a eee 
Of. o8.¢. | 8e8.0\) “FEY. 

08.€ 68.0. 882.9 and 
0d. BOB.0.) 800.0: ie 
08.£ 038.0 ee. yak 
og fe 808.0. 2td.0~ a 
62,4 21820 SFe.0 | ag Oe 
Ox. $88.0 OOF 0 a 
ore . | | OV8s8 Ot8.0 i = Gemet i 
ove tVt.0 OSH.0: 1 eo 
60.4 |» B¥8.0 Sie +0 Af a 
o8.o | 698.0 © 2#8,0 eee 
ase 4 “Bdt.0 erh.a eo ie 
OO.€  {' #AB.0 20050" fe RAD vi 
OB.E ASB.0 ro oe | Ne 
Odie, 4) OBN,0 4 Beto | ue / 
09. f08.0 8 §= fe 20, ome” 
oe. i eee a 2) ee ae 
Os.4 (34,0 dee.0 || 82) 0 ae r 
08.) 88.6 $2.0 | @ ee 
OS.9 {8,0 _  O&2.0 TA: 
OS oP FO | | ORES S Pe ae oe 
of. aie (€%.0 ae ee 
oid £08.90 tae Aas ae 
0° .S eo % 1 888,90 oan a 
00.€ abajo t0s,0: | ay S ea 
oye | andi ie oe ye: as 
ee X89. Oo et ve ia, 
Of.# | 888.0, o | \ Ging ze | 
Od.¢ ef9.0— a | 
02.2 @fa.0 VSO) 
02.4 vt ; 
OS.2 >} Cad.d " BE 2 ae) 
oe | 8880 ie es Ge | 
00.€ |. €20.9 On ae oe) 
Of. po) Bd.) 1 18 7. 
Ose | 088.0 \ Ve er ae 


Jody! vd henieido zoutey ds ssiwionie eoren 16, aa1oo2 euad os Foo hll 


, 


138. 


(a) The non-normal latent score distributions affect the test 
Parameters such as variance, reliability and KR20, and with lesser 
degree the mean, if the distribution is skewed. The normal ogive 
model provides smaller values than the actual values of the variance, 
reliability and KR20 if the latent scores are distributed as uniform, 
and the opposite is true for exponential distribution. 

(b) Formulas (5.1) and (5.3) are quite robust against the 
violation of assumptions of normality for the binary item score 
cases. 

(c) The effects of item difficulty parameters are the same as 
observed in the previous section. 

(d) The non-normal latent scores do not systematically inflate 
or deflate real Type one errors for the F-test. The F-test seems 
quite robust against the violation of distributional assumptions, if 


difficulty parameters are homogeneous. 


6.4 Effects of Non-Homogeneous Biserial Correlations 


For the previous two sections, the biserial correlations 
were limited to homogeneous cases, namely Bl, B3, and B5. In this 
section, three non-homogeneous biserial correlation sets, B2, B4, and 
B6 are used to investigate the effects of such non-homogeneity. Since 
it is known that for the continuous part-test score cases the non- 
homogeneity of true score variance, which corresponds to the square of 
biserial correlation for the binary item case under the congeneric 
true score model, does not affect the sampling distribution of the 
reliability estimates if the non-homogeneity is moderate, and it is 


of interest to know whether the same conclusion can be made for the 


binary item cases. 
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Employing three kinds of latent score distributions, Ul, 
NO, and EX, and four sets of difficulty parameters, as in the 
previous section, and three sets of non-homogeneous biserial correlation 
sets, altogether 36 (3 x 4 x 3) additional experiments were performed 
with N= 1000, I = 30, and J = 9. The results are summarized in 
Tables 6.9, 6.10, and 6.11. 
If the test parameters estimated by the parallel form method 
in Table 6.9 are compared with the corresponding entries of Table 6.6, 
the latter table using the same parameter distribution combinations as 
in this section except that the biserial correlations are not homog- 
eneous , although the averages of the biserial correlations are the same, 
it is noted that the results of the two tables are almost identical. 
This suggests that the effects of non-homogeneous biserial correlations 
are small, even though the non-homogeneous biserial correlations do 
violate the ETEM assumptions, and consequently lower the KR20 relative 
to the reliability. 
Although the biserial correlations are not homogeneous, 
almost the same conclusions may be made for Tables 6.10 and 6.11 as for 
Tables 6.7 and 6.8 respectively; 
(a) the means and standard errors of reliability estimates are 
almost identical in the two sets of the experiments, 
(b) the F-tests are quite robust against the violation of the 
ANOVA model and normal distribution theory for the binary item cases 
if difficulty parameters are homogeneous, and 
(c) the item difficulty parameters affect the distribution 


considerably, if they are not homogeneous, thus inflating or deflating 


real Type one errors for the F-tests. 
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TABLE 6.9 


Comparisons of Calculated Test Parameters Under the Normal Ogive Model With 
Empirical Values Based on the Parallel Form Method, Normal Error Scores, 
Non-Homogeneous Biserial Correlations, NI = 30030, J =9 


Theoretical (N.0.) Observed by P.F.M. 
KR20 Rel. 
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TABLE 6.10 


Comparltsons of Observed Means and Standard Errors of Reliability Estimates 
Under Normal Error Scores and Non-Homogeneous Biserial Correlations 
With the Values Obtainable From ANOVA Model and Nga | ? 
Theory, N =+1000;.«l = 30, J =.9 


Observed a. yb cts SE by 
Mean G- 1st if 3) (5e})- (b) 
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TABLE 6.11 


Comparisons of Observed Lower and Upper 5% Critical Points Under Normal Error 
Scores and Non-Homogeneous Biserial Correlations With the Values 
Obtainable From the ANOVA Model and Normal Theory, and Real 
Type One Error of F-Test When Nominal Value is Fixed 
to the 5% Level, N= 1000, | = 30, J=9Q 


Observed C.P.7 Theoretical €.P.°| Real Sig. (0/0) 


Upper Upper Lower Upper 
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From the above observations, the following conclusions are 
tentatively made. 

(a) The non-homogeneous biserial correlation distorts the 
distribution slightly to the left for high reliability cases, but the 
differences are not substantial. 

(b) The effects of non-homogeneous biserial correlations on 
expected values and standard errors of reliability estimates are 
minimal, and formulas (5.1) and (5.3) are quite satisfactory. 

(c) The effects of non-homogeneous biserial correlations on 
test parameters are minimal. 

(a) The F-tests are robust for binary item test cases if the 


difficulty parameters are homogeneous. 


6.5 Summary 


In order to investigate the effects of non-normal latent and 
error scores, non-homogeneous difficulty parameters and biserial cor- 
relations on the sampling distribution of reliability estimates based 
on formula (2.13), altogether 96 experiments were performed by RELO2 
using various combinations of distribution parameter sets with N = 1000, 
| = 30, and J = 9. The findings in this chapter may be summarized as 
the following: 

(a) The effects of non-normal distribution of error scores fei} 
in terms of response strength variables yz} are negligible, as was 
the case for the continuous part score cases in Chapter Five. 

(b) The non-normal latent scores affect the population para-~ 
meters such as variance, reliability and KR20. The normal ogive 
model underestimates these parameters for the uniform latent score 


distribution, and overestimates them for the exponential case. 
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(c) Formulas (5.1) and (5.3) are quite satisfactory for binary 
item cases; formula (5.1)-(b) seems superior to (5.3) for the calculation 
of the standard error of reliability estimates. 

(d) The item difficulty parameters are the most important factor 
for the distribution of reliability estimates. They will affect the 
test score variance, reliability and KR20. The non-homogeneous difficulty 
sets give lower values for these parameters. 

(e) The item difficulty parameters systematically affect the 
distribution of reliability estimates. The non-homogeneous difficulty 
sets shift the distribution leftward. 

(f) The effect of non-homogeneous biserial correlations are 
eo tease if the heterogeneity is moderate. 

(g) The F-test based on (2.17) is robust if any one of the 


following conditions is satisfied. 


i) Relibability is low, i.e., po is close to zero. 

ii) Only lower portions of the sampling distribution of 
reliability estimates are used for the inference, namely 
the null hypothesis is directional, being bounded only 
by the lower end. 


iii) The difficulty parameters are almost homogenous. 


(h) The difficulty parameter sets may deflate the real Type one 
errors if they are not homogeneous for inference which uses only upper 


tail of the sampling distribution. 
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CHAPTER SEVEN 


SUMMARY, IMPLICATIONS, EXAMPLES OF APPLICATION 


5 


AND RECOMMENDATIONS 


etl sO Summary of Findings 


The purpose of this study was (a) to review the more liberal 
concepts of test reliability theory in terms of models and assumptions 
underlying them, (b) to examine the sampling distribution of reliabil- 
ity estimates based on Alpha or KR20 formulas using these models with 
various combinations of the distribution of true and error scores, and 
(c) to compare the empirical distributions thus obtained by computer 
simulation under these model-distribution combinations with those 
obtainable theoretically under a mixed model ANOVA and normal theory 
Ustng computer simulated hypothetical test score matrices, a number of 
statistical sampling experiments were performed to obtain empirical 
distributions, and some analytical means were also employed to obtain 
a new formula for the standard error of reliability estimates. Find- 


ings in this study are summarized in the following three sections. 


7.1.1 Test Models 


(a) The most general model for the continuous part test score 
is found to be the multi-factor true score model. The model includes 


other more restrictive models as special cases. By imposing a uni- 
factor true score constraint, the model becomes a congeneric true score 


model lf homogeneity of true score variance is assumed, the congeneric 


model becomes essentially t equivalent measurement. The latter model 


includes the ANOVA or essentially parallel measurement model as a special 
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case with the additional assumption of the homogeneity of error 
variances. The classical parallel test model is a special case of 

ANOVA model, namely the means of part test scores are all equal. The 
Alpha coefficient is equal to the reliability if, and only if the 
essentially +t equivalent measurements condition is satisfied, other- 
wise it is in general lower than the reliability. The sampling distribu- 
tion of reliability estimates is known only for the case of the ANOVA 
model and normality assumptions of true and error scores. 

(b) For the binary item test pa a similar model as the 
continuous case may be considered for the hypothetical 'response 
strength' variable. A mathematical model and distributional assumptions 
Bes sine to associate the response strength variable to the observed 
item scores. Under the normal ogive Senet the test parameters such 
aS variance, reliability, and KR20 are amenable for calculation by 
means of numerical methods if the item parameters, such as biserial 
correlation and difficulty parameters, are specified. The essentially 
t equivalent measurement assumption is satisfied if and only if all 
biserial correlations and difficulty parameters are equal, i.e., al] 
items are homogeneous; otherwise KR20 is lower than reliability. The 
sampling distribution of reliability estimates for binary item test is 


not yet known, except by approximation using the ANOVA model and normal 


theory. 


Delage Sampling Distribution Under Various Models and Assumptions 


(a) Applying Tukey's result, a new formula for the standard 
error of reliability estimate was derived. The formula depends only 


on sample size, number of part tests, reliability, and the kurtosis 
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of the test scores, and is found to be superior to the traditional 
formula based on normal theory when the distribution of true score is 
not normal. 

(b) The effects of non-normal error scores distributions are 
found to be negligible for not too small J, the number of part tests 
or items, for both the continuous and binary cases. 

(c) For continuous test score cases, the effects of non- 
normal distributions of true scores are found to be significant, i.e., 
the distribution of reliability is systematically distorted. If the 
essentially t equivalent assumption is not satisfied, the distribution 
is systematically shifted leftward or to the lower direction of 
reliability. This effect is moreclearly observed for the multi-factor 
true score model case indicating inappropriateness of the Alpha formula 
for the model. The effects of non-homogeneous error variance were found 
to be negligible. 

(d) For the binary item case the effect of non-normal dis- 
tributions of latent scores is not so obvious. The formula for the 
standard error derived under the ANOVA model and normal theory seems 
quite robust against violation of assumptions imposed by a binary 
scoring scheme. The test parameters depend on the shape of latent 
score distributions for fixed biserial correlation and difficulty 
parameters. If the essentially t equivalent measurement assumption 
is not satisfied, i.e., biserial correlation and/or difficulty 
parameters are not all homogeneous, the distribution of reliability 
estimates is shifted leftward systematically. The effects of non- 


homogeneous difficulty parameters seems more severe than that of non- 


homogeneous biserial correlations. 
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7.1.3 Robustness of F-Test 


The F-test based on ANOVA model and normal theory is robust 
against violation of the following assumptions: 
(a) Normality of error scores for both continuous and binary 
cases. 
(b) Homogeneity of error score variances for continuous cases. 
(c) Homogeneity of biserial correlations for the binary case if 
the violation is not too extreme. 


(d) Normality of latent score distributions for binary case. 


The F-test may be misleading if the following conditions are 

not satisfied. 

(a) Uni-factorness of true and latent score distributions. 

(b) Normal ity of true scores for continuous case. Especially 
positive kurtosis of true scores results in severe distortions. 

(c) Essentially t equivalent assumptions (approximately at 
least). 

(d) Homogeneity of item difficulty parameters (approximately at 


least). 


Nevertheless, in all cases, the F-test is robust against any 
violation of assumptions if the population reliability is close to zero 
for both continuous and binary cases. If only the lower tail portion 
of the distribution is used for the binary item test, the significance 


test is also robust in most cases. 


7.2.0 Implications to Test Theory and Applications 


In this study, it has been demonstrated that the distribution 


of the reliability estimate depends significantly on the models employed, 
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the underlying assumptions, and the parameters of part tests or items. 
Therefore the validity of any statistical inference about reliability 
largely depends on the validity of models and assumptions like any other 
statistical inference. Therefore it is essential, for statistical 
inference about reliability, to know the models appropriate for the 
test in use, and the population characteristics for the test must be 
known a priori. For a casual user of psychological and educational 
tests, this is an almost impossible task. Therefore, for test users 
and/or other researchers, the findings in this study may not be of any 
practical use without knowledge of the above information about the test 
except when robust conditions are present. 

However, for a test author, or for a test reviewer, the task 
of gathering the necessary data may be accomplished as a by-product of 
the usual procedure for the test development, since an administration 
of the test to a comparatively large sample of subjects from the 
population for whom the test is developed is usually involved in order 
to standardize and to obtain test norms. The test statistics based on 
such large samples may be used to obtain such information. 

Although there is no agreed upon statistical and psychometric 
methods to obtain such parameters, some efficient methods for the 
calculation of part test parameters have been developed recently by a 
number of psychometricians. 

For example, Kristof (1969) considered the estimation of the 
and error score variance {on} under an 


e 2 
true score variance 0, 


essentially +t equivalent measurement assumption by employing maximum 
likelihood method. He derived the likelihood equations and found that 
these could be solved rapidly by a simple Newton-Raphson procedure. 


For the binary item test cases, the item difficulty is easily 
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calculated, and the biserial correlation parameters may be obtained 
by factor analysis of the tetrachoric correlation matrix from the 


results of (3.14), if the latent score has a uni-factor structure. 


Jéreskog (1971) has shown some examples of model identifica- 
tion techniques by employing maximum likelihood factor analysis on the 
disperson matrices of test scores obtained from large samples. 

In regard to distributions, the distribution of error score 
is found to be not important, but the shape of the distribution of true 
scores can affect the reliability estimate significantly. Although 
the distribution of true scores is not observable directly, since only 
the kurtosis of true score will affect the distribution of reliability 
estimates, and it can be indirectly evaluated by the test score 
kurtosis divided by the square of reliability from the results of (5.9), 
the normality of true score may be investigated partly by examining the 
test score kurtosis if it is obtained from a large sample. 

Therefore, a test author or reviewer would be doing a service 
to the users of a test, if he provided information about the model 
involved, and distributions and parameter values in the population for 
which the test is developed. If the test satisfies the ANOVA model and 
normal theory assumptions, or violates only those assumptions which are 
known to be unimportant, the author or reviewer may recommend the 
use of the F-test for the inference about reliability. In this case 
the author or reviewer needs to supply only the information about the 
population reliability. Otherwise, the author or reviewer should either 
provide all information necessary for simulation of such tests by the 
computer program developed in this study, or alternatively, provide a 


table of upper and lower critical points of the distribution of 
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reliability estimates as a function of sample size, and should probably 
also provide the values of the standard errors. Then the user could 
easily determine whether the observed reliability is significantly 
different or not from the population value at a specific significance 


level. 


7.3.0 Example 1: Application to Continuous Case 


Since it was not possible to find an appropriate example of 
a test and its manuals which provide the necessary information for the 
test models and the other information necessary for the application of 
computer simulation techniques, somewhat arbitrary example data were 
selected to show how the findings in this study and the computer 
programs developed might be applied in a practical situation. 

Jéreskog (1971) analyzed a dispersion matrix based on four 
measures used by Votaw (1948) to establish methods of obtaining reader 
reliability in essay scoring for an English composition test, and 
identified the model as a congeneric true score model. The dispersion 


matrix was obtained from 126 subjects, and is given in Table 7.1. 


TABLE 7.1 


Dispersion Matrix of Votaw's Essay Test Data, 
| = 126 


Measure 
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He employed maximum likelihood factor analysis, and gave the 


estimate of the standard deviation of true score or factor loading as, 
Nicer 57) 2.68 2.65)4. 53) 2 


Therefore, if a test author published a test consisting of 
four part tests and obtained the same results as above based on a large 
sample, these values may be regarded as population parameter values if 
Then the test 


small discrepancies in covariance terms are ignored. 


score model would be as follows, 


4.57 fF 2.0459 0.0000 0.0000 0.0000 Bj 
2.68 , | 90-0000 4.5847 0.0000 0.0000 
Wee to” GS 0.0000 0.0000 3.9644 0.0000 
453 0.0000 0.0000 0.0000 1.1618 
Then, 
25.0704 12.2476 12.1105 20.7021 
" ; 2 | 12.2476 28.2021 7.1020 12.1404 
Dly,) =Z=AA'+¥ = | 5 't105 7.1020 22.7390 12.0045 |» 
20.7021 12.1404 12.0045 21.8707 
and, 
Alpha = 0.812329, o = 0.831249. 


Since the assumption of the homogeneity of true score 


variances is violated, the essentially 


t equivalent measurement 


assumption is not valid and hence the Alpha coefficient is lower than 


reliability as expected. 


From the findings of this study, it is known that the effects 


of non-homogeneous true score variance is not too great with moderate 


differences among the elements of the factor loading vector A , 


but 
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the differences for this data seem exceptionally large and also the 
difference between Alpha coefficient and the reliability is substantial. 
Therefore a systematic distortion of the distribution of reliability 
estimates toward lower reliability is expected. Seven computer 
simulation experiments were performed with the J&reskog's model with 
N = 2000 and assumed normality of true and error scores. Both estimation 
formulas, namely the Alpha formula of (2.13) and Kristof's unbiased 
formula of (5.2)-(a) were used for estimation of sample reliability. 
Observed upper and lower 5% critical points together with standard 
errors are summarized in Table 7.2. The observed values are also 
compared with those obtainable under the ANOVA model and normal theory. 
The sample sizes |, the number of subjects, used for these experiments 
are 10, 15, 20, 25, 30, 35, and 40 respectively. Figures 7.1 - 7.7 
compares empirical distribution with the theoretical distributions 
indicating the effect of the violation of the essentially +t equiv- 
alent measurement assumptions. 

A table similar to Table 7.2 might accompany the test 
manuals or test review report so that test users may consult the 
table whenever they make inferences about the reliability. For 
example, if a teacher administered the test to a sample of 20 students 
and obtained 6 = 0.892, then by consulting this table she may conclude 
that the difference between the population value 0.812 and her sample 
value is not significant at 5% level of Type one error. Therefore, 
she may not claim that her sample is significantly different from the 
population for which the test is developed as far as the reliability 
The author or researcher could develop a slightly 


is concerned. 


different table if the population test score is not normal. For 
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FIGURE 7.1 V@TAW-JORESKOG DATA, N=2000, [=10, NORMAL 
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FIGURE 7.2 VOTAW-JORESKOG DATA, N=2000, J=15, NORMAL 
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FIGURE 7.3 VOTAW-JORESKOG DATA, N=2000, I=20, NORMAL 
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FIGURE 7.4 VOTAW-JORESKOG DATA, N=2000, I=2S, NORMAL : 
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FIGURE 7.5 VOTAW-JORESKOG DATA, N=2000, 1=30, NORMAL 
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FIGURE 7.6 VOTAN-JORESKOG DATA, N=2000, 1=35, NORMAL 
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FIGURE 7.7 VOTAN-JORESKOG DATA, N=2000, I=40, NORMAL 
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example, if ranked marks were assigned for each part test, the greater 
likelihood is that the true scores would be distributed uniformly 


rather than normally, and the shapes of reliability estimates would be 


much different. 


Tan.0 ‘Example 2: Application to Binary Item Case 


A hypothetical binary item test consisting of 9 items is 
considered as an example. The values of item parameters are taken 


from Lord and Novick (1968, p. 379), and summarized in Table 7.3. 


TABLE 7.3 


Item Parameters of a Nine Item Test 


Difficulty | 0.096 0.199 0.338 0.434 0.471 0.574 0.676 0.801 0.822 


Biserial 0.490 0.717 0.549 0.593 0.595 0.640 0.476 0.530 0.495 


Cor. 


It may be noted that the item difficulty parameters are rather 
heterogeneous with a value as small as 0.096 to as high as 0.822. There- 
for it is expected that the essentially t equivalent measurement 
assumption is substantially violated. To investigate the sampling 
distribution of reliability estimates of a binary item test with these 
parameters under the normal ogive model, an experiment was performed 
with | = 30, and N = 1000. 

The theoretical test parameters and those obtained by 


parallel] form method are compared in Table 7.4. 
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TABLE 7.4 


Test Parameters of a Nine Item Test 


4.4710 4.0054 0.6632 0.6498 
44774 h 0023 0.6654 0.6526 


Therefore a user of this test may compare her observed test 


Methods 


Theoretical 


Parallel Form 


mean, variance, and KR20 with the values given in this table, and can 
make some conclusions about her sample group. 

The shape of the distribution of reliability estimate based 
om (2.13) js compared with the theoretical distribution under the 
ANOVA and normal theory model in Figure 7.8. The distribution shows 
a systematic shift leftward probably due to heterogeneous difficulty 
parameters. The lower and upper 5% critical points of this distribu- 
tion are 0.4412 and 0.7793 respectively while the theoretical values 
are 0.4466 and 0.7656 respectively. Therefore, if a user of the test 
found a reliability estimate of 0.79 with | = 30, it may be concluded 
that the reliability is significantly higher than the population value 


at the 5% level of significance. 


7.5.0 Recommendations 


As noted in Section 4.8 of Chapter Four, in the discussion 
of the methodological limitation of this study, the computer simulation 
experiments cannot be exhaustive and cover all possible combinations 
of models, parameters, and distributional assumptions. Also due to 


the limitations imposed by limited funds available for the computing 
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FIGURE 7.8 LORD-NOVICK DATA, N=1000, I=30, NORMAL 
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charges, the scope and extent of experiments have been restricted to 
certain special cases which may not always be directly relevant to 

real data. Because of these facts, the findings of this study will 

be limited to some extent in their generalization and application. 
Therefore, the findings will, by circumstance, be exploratory and 
illustrative rather than comprehensive with the emphasis having been 
placed on methodology. Based on the findings and experience with the 
computer simulation techniques, the following recommendations are made: 

(a) The computer simulation techniques can be used to solve 
many statistical and psychometric problems in test and measurement 
theory and application. Further use of this technique is recommended 
and research must be carried out to improve the methodology. 

(b) Authors of published tests, or their reviewers, should 
attempt to specify the appropriate test model for a given test, and 
place such information in the test manuals. The manuals should also 
include the population dispersion or tetrachoric correlation matrix 
of true or latent scores or estimate of them as well as the parameter 
values such as error variances, difficulty and biserial correlations 
based on a large sample. The distributional characteristics of true 
or latent scores and error scores of the population for which the test 
is developed should also be included. 

(c) Some of the findings in this study are based on only 


a few parameter sets and distributional assumptions. Therefore, the 


findings must be confirmed by replicated studies with 


a wider range of parameter sets and with distributions of different 
shapes of true or latent and error scores, and if applicable, using 


real test score data. More specifically, the following aspects require 
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further investigation: 

(i) The effect of non-homogeneous true score variance, i.e., 
the distribution of reliability estimates under the congeneric 
model for a wider range of peer 


(ii) The effects of non-homogeneous item difficulty parameters 


for the binary item test cases. 


(d) In this study, the size of sample is artificially fixed 
at | = 30. An investigation must be made to examine the effects of 
sample size to see how fast the estimate converges to its expected value 
with increasing sample size. 

(e) The investigation of this study was limited to Type } 
sampling situations only, but a similar method can be employed for Type 
2 or Type 12 sampling situations possibly with a different type of 
ANOVA model. 

(f) The test score used in this study was a simple unweighted 
sum of J part test or item scores, although a weighted sum could have 
been easily employed. The effects of a weighted sum on the reliability 
estimate must be explored as an extension of this study. 

(g) In this study, one of the basic assumptions of test 
theory was assumed to be always valid. The assumption was one of 
independence of error and true scores. In practice this may be 
violated and the effects of such violation on the distribution of the 
estimate of reliability must be investigated. Computer simulation would 
provide an ideal method for such an investigation. 

(h) It is clear that Alpha coefficient as an estimate of 


reliability is inappropriate if the essentially t equivalent 
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measurement assumption is violated too much. Therefore a new effort 

is necessary to find an appropriate means to estimate reliability under 
this condition, especially for the case of the multi-factor true score 
model. 

(g) In this study, the investigations were limited to one 
sample and one reliability estimate cases, and comparisons of the 
estimate to the population value, but similar methods may be applied 
to investigate for the cases of more than one sample or reliability 
estimates either based on independent samples or repeated measures on 
the same sample to investigate the sampling distribution of the 


differences of the reliability estimates. 
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APPENDIX A.1 


LISTINGS OF COMPUTER PROGRAMS 


RELO] : Simulation Program for Continuous 


Part Test Case 


RELO2 : Simulation Program for Binary Item 
Test Case 
RELOO : A Package of Sub-Programs Shared by 


RELO] and RELO2 
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FORTRAN IV C€ COMPILER MAIN 09-15-71 153:55.55 PAGE OOOL 


RELOL DIVISICN OF EDUCATIONAL RESEARCH SERVICES 
UNIVERSITY OF ALBERTA 
ZOCOR ICOR SRO RO tak Sack a FOR ORR OI took RO OR Roi iotetoiog irk kr tok glak tot tek tok ge ge 
PURPOSE: STIMULATES CCNTINUOUS PART-TEST SCORES BASED ON MULTI- 
FACTOR TRUE SCORE MODEL TO INVESTIGATE SAMPLING 
DISTRIBUTION OF RELIABILITY ESTIMATES 
CARD INPUT: 
le TITLE(20A4) 
2. PARAMETERS(11LI55F5.5) SNSAMy MI »MJgNFoIX,IDIST,IDISE, 
IPUNCK, IPLOT,MOCE,LB,y SIG 
NSAM NO OF SAMPLES SIMULATED 


MI NO OF SUBJECTS IN THE SAMPLE 

MJ NO OF PARTS 

NF NO OF FACTORS IN TRUE SCORE 

IX ANY OOD INTEGER TO INITIATE RANDOM NUMBER 


IDIST OPTION FOR THE OISTRIBUTION OF RANDOM 
EFFECTS (TRUE SCORE) 
O-NORMAL 
1-SPECIFIED BY SUBPROGRAM DIST 

IDISE OPTION FOR THE DISTRIBUTION OF ERROR 
O-NORMAL 
1-SPECIFIED BY SUBPROGRAM DISE 

IPUNCH OPTION FOR CARD OUT FUT OF FREQUENCIES 
O-NO CARD OUTPUTS 

1-CARD CUTPUT REQUIRED 

IPLOT CPTION FOR PLOTS 
O-NOT REQUIRED 
1-REQUIRED 

MODE OPTION FOR ESTIMATICN FORMULA 
O-ALPHA FORMULA( BIASED) 
1-KRISTCF CORRECT ION(UNBIASED) 
2-BOTH OF ABOVE 


LB OPTION FOR THE NO OF CLASS INTERVALS FOR 
THE FREQUENCY CALCULATION »24,36 OR 48, 
i ASSUMED 24 
sre SIGNIFICANCE LEVEL FOR EACH TAIL ,ASSUMED 
0.05 
3. FMT FORMAT FOR THE INPUT VECTORS AND MATRIX 
4. FIX A VECTOR OF MEANS FOR EACH PART 
5. ERR A VECTOR OF STANDARD DEVIATION OF ERROR 
SCORES FOR EACH PART-TEST 
6. FAC A FACTOR LOADING MATRIX OF SIZE MJ BY NF 
7. A BLANK CARD 
REMARK: 
1.CURRENTLY DIMENSIONED TO ACCOMODATE UP TO FOLLOWING 
SIZE PARAMETERS 
NS AM 5000 
: MI 100 
MJ 30 
NF 10 
LB 48 


SUBPROGRAMS?: { FORTRAN) ANOV,BOXSN,CHI PRB, COUNT DI SCRP ,01SPy EXAMPLs 
FISHER »FITTESs FST» MXOUT, PLOT »POPR» PUNCHyRELDIS,ROZBy 
SIGTES»VARXX,VECRAN,VEOUT, DATA,DIST,DISE 


AANAAANQNANAARAAAAARAAAAAARAARAAARAARAARAARANAMAAARANAAKRAQNANKRAARNANAANRAARAaAANNAAAA 
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FORTRAN IV G CCMPILER MAIN 09-15-71 15255.55 PAGE 0002 


0ool 


0002 
0003 
0004 
0005 
0006 
0907 


0008 
0009 
0010 


OOllL 


0012 
0013 
0014 
0015 
0016 
O01? 
0018 
0019 
9020 
0021 
0022 
0023 
0024 
0025 
0026 
0027 
0028 
0029 
0030 
0031 
0032 
0033 
0034 
0035 
0036 
0037 
00 38 
0039 


Cc (*SSPLIB) BOTR»COTR»,DLGAM,NDTR y RANK 

C PROGRAMMER: Ke BAY 

Cc NMAX=LARGER OF NSAM AND MI#MJ 

c DIMENSION FAC(MJ*NF) sERR( MI) »pVAR (MJ) pDIS( MS 9 MJ) »FIX(MS) »BSUMO(MJ) » 
C LBSS(MJ)yY(MI*MJ),BB(MJ) »FMS(NS AM *3),FREQ(LB*3) » TEMP(MI*NE) 


DIMENSICN TITLE 20),FMT(20),LAB( 4),FAC( 300),ERR( 30),VARI 30), 
LOTS( 900),FIX{ 30),BSUM( 30),BSS( 30),Y{ 3000),8B( 30),FMS( 15000) 
29FREQ(144) ,XBAR( 6)yXVAR(I 6), TEMP( 1000) 

REAL*8 LAB 

CATA LAB/*SUBJECT*",*PARTTEST*, "ERROR*, "REL COF®/ 

100 FORMAT(20A4) 

101 FORMAT(LKL,20A4) 

102 FORMAT(1115,F5.5) 

103 FORMAT{/,10X»*NO OF SAMPLES SIMULATED® 515X» I14¢/,10X,*°NO OF SUBJEC 
1TS IN EACH SAMPLE® ,11X_9Il29/910Xy"NO OF PART-TESTS® »24X,129/,10X, 
2°NO CF FACTORS IN TRUE SCORE*»)14X¢9I159/7910Xs*STARTING INTEGER RANOO 
3M NUMBER* 5 2X,1109/,10X,*OPTION FOR CARD OUTPUT® 419XeI1le/y1LOX, 
4*OPTION FOR PLOT®%,26X,I19/,10X,* OPTION FOR ESTIMATION FORMULA’, 
512XeT1lys/10Xy*OPTICN FOR THE NC OF CLASS INTERVALS * »3X,139/y,10Xy 
6"*SIGNIFICANCE LEVEL® ,19X,F5.3,/) 

104 FORMAT(2X%_ 1345X_ 2E 14 0692X9 "4 %3 2X9 2E14.6) 

105 FORMAT(/51X, **e¥x****XLAST SEED RANDOM NUMBER [X=*,I110) 

106 FORMAT(/,1X,*DISCRIPTIVE STATISTICS FOR FIXED EFFECT ESTIMATES AND 
1 EXPECTED VALUES UNDER M.Fe MODEL *,s/,1X_e*PART* ys 7X9 *MEAN% yg LOX,* EXP 
ZECTED' ,6Xy*]%,5X_* VARIANCE *,8Xy5* EXPECTED® ) 

LOT FORMAT( LHL 9230 'Q") pJ/olLXy*D*y2X_o" SUMMARY OF CUTPUT® 92Xe*d"y/e1X,23f 
1l*a*)) 

108 FORMAT(1LOX,*ERROR SCCRE OISTRIBUTIGNS ARE NORMAL*) 

109 FORMAT(1LOX,*ERROR SCCRE DISTRIBUTIGNS ARE NOT NORMAL*) 

110 FORMAT(10X,"*TRUE SCORE DISTRIBUTIONS ARE NORMAL ® ) 

111 FORMAT(LCX,*TRUE SCORE DISTRIBUTIONS ARE NCT NORMAL® ) 

10 REAC(5,100) TITLE 

IF(TITLE(1)-EQ-TITLE(2)) GO TO 99 

WRITE(6,101) TITLE 

READ(5,102) NSAMgMIyMJgNFo IXeIOIST,IO1SE, I PUNCH, IPLOT,MODE,LB,SIG 

LF(SIGLL. LE-0.0) SIG=0.05 

IF (LBeNE.24 eANDeLBeNEo 36eANDoLB2NE48) LB=24 ~ 

IF(NFeLE.O) NF=1 

WRITE(65103) NSAMsMI pMJo NF 9 IXe IPUNCHy I PLOT » MODE,LB,SIG 

IFC IDIST.EQ.0) WRITE(6,110) 

IF(IOIST~EGe1) WRITE(6, 111) 

IF(IDISE.EQ.0O) WRITE (6,108) 

IFC IDISE-EQe1lL) WRITE (6,199) 

CALL PCPR(MJ »NF 9 FAC, ERR» VAR, DIS» REL» ALPHA, TVARyEVARs FMT pF IX» GMEAN) 

TVAR=TVAR/ (MJS*MSI) 

EVAR=EVAR/MJ 

THETA=TVAR/EVAR 

DIV=1l-OtNJ*THETA 

CO 20 J=1l,™MJ 

BSUM(J)=0.0 

20 8SS(J)=0.0 
CALL VECRAN(Y,300, IX) 
CALL EXAMPL(MI »MJ~)NF sFACeERReLOIST pI OLISEs IXsV¥yF IX, TEMP, BB, FMS,REL) 


pO 50 NTRIAL=L»NSAM 
CALL DATAUMI MJ gNF pFACe ERR eg IDIST pI DISEy 1 Xs V¥ oFIXs TEMP) 
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FORTRAN IV C CCMPILER MAIN 09-15-71 1§:55.55 PAGE 0003 
0040 CALL ANOV(Y»)MI »MJ_yFMSA,FMSBy EMSE, BB) 
0041 FMS(NTRIAL)=FMSA 
0042 ITT=NTRIAL4+NSAM 
0043 FMS(IL)=FMSB 
0044 IT=114+NSAM 
0045 FMS(II)=FMSE 
0046 CC 45 J=1,¥J 
0047 BSUM(J)=BSUM(J)+BB(J) 

0048 45 BSS(J)=BSS(J)+B8B (J )* #2 
0049 50 CONTINUE 
0050 DO 54 J=1,MJ 
0051 54 FIX(J)=FIX(J)-GMEAN 
0052 WRITE(6,107) 
CALL MXOUT{FMS sNSAMy 340144,44HMEAN SQUARES: COL-1 MSA,COL-2 MSB,CO 
0053 NN=NSAM*3 
0054 XMAX=0.0 

0055 CO 56 I=1,NN 

0056 56 IF(FMS(I).GT«XMAX) XMAX=FMS(1) 

0057 NN=XMAX/10.041.0 

0058 XMAX=NN*10 20 

0059 TINT=XMAX/LB 

0060 CALL COUNT (FEMS »NSAMy 39020, TINT gL By FREQ yXBAR pXVAR) 

0061 XMIN=0.0 
0062 XMAX=TINT*LB 
0063 XBAR(4) =MJ*TVAR#EVAR 
0064 XXXX=0 20 
0065 DO 58 J=l,MJ 
0066 58 XXXX=XXXX+FIX( J) eZ 
0067 CFA=MI-1 
0068 DFB="J-1 
0069 DFE=(MI-1)*(MJ-1) 

0070 XX XX=XXXX/DFB 
0071 XBAR(5) =EVAR+XXXX¥MI 

0072 X8AR(6)=EVAR 
0073 XVAR (4) =(2.0%(MJ*TVARFEVAR ) #2 )/ DFA 
0074 XVAR(5)=(( EVAR42 .O*%MI*XXXX )*2.0*EVAR)/OFB 
0075 XV AR (6) =(2 -OXEVAR*EVAR) /OFE 
0076 CALL DISCRP(3,XBAR »XVARyLAB(1)752952H MEAN SQUARES AND EXPECTED VA 

LLUES UNDER ANOVA MODEL ) 
0077 CALL VARXX(NSAMyMJ »BSUM,BSS) 
0078 WRITE(6,106) 
0079 DO 63 J=lyhJ 
0080 XX=0.0 
0081 DO 61 M=lyMJ 
0082 01=-1.0/¥J 
0083 IF(JeEQeM) D1=D1+1.0 
00 84 DO 60 K=ly¥J 
0085 02=-1.0/MJ 
0087 MK=N Je (M-1L)4K 
0088 60 XX=XX+DL*D2¥*O01S (MK) 
0039 61 CONTINUE 
0090 XX=XX/MI 
0091 63 WRITE(6,104) JeBSUM( JS), FIX( I), BSS(I) 9XX 
0092 LI=NSAM¥®2 
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FORTRAN IV € COMPILER MAIN 09=15=72 15255.55 PAGE 0004 

0093 NO 70 [=1,NSAM 

0094 Il=1I+l 

0095 70 FMS(1T)=1.0-FMSCII)/FMS(T) 

0096 CALL RANK{FMS(1),FMS(NSAM4#1),NSAM) 

0097 IF (MODE.NE«1) CALL RELDIS(FMS,NSAM,DFA,DFE »FREQsLB,REL»TEMP SIG, 
~ LXBAR»,XVAR» IPUNCHy IPLOT,0,LAB) 

0098 IF(MODE.NE~O) CALL RELDIS(FMS,NSAM,DFA, DFE, FREQsLBeREL » TEMP s SIG, 

LXBARyXVAR»,IT PUNCH, IPLOT,1,LAB) 

0099 : WRITE(6,105) IX 

0100 GO TO 10 

9101 99 STOP 

0102 END 


TOTAL MEMCRY REQUIREMENTS 015284 BYTES 
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FORTRAN IV CG COMPILER CATA 09-15-71 15:56.00 PAGE 0001 
0001 SUBROUTINE DATA(MI sMJsNFo FAC, ERR» IDIST, IDISEyIX) VyF IX» TEMP) 
C PURPGSE CREATES DATA MATRIX FOR RELOL 
Cc MI SAMPLE SIZE 
C WJ NO OF PARTS 
Cc NF NO OF FACTORS’ IN TRUE SCORE 
C FAC INPUT FACTOR LOADING MATRIX 
Cc ERR INPUT VECTOR OF STANDARD DEVIATION FOR ERRORS 
Cc IDIST OPTION FOR TRUE SCORE OISTRIBUTION 
Cc IDISE OPTION FOR ERROR SCORE DISTRIBUTION 
Cc IX SEED ODD INTEGER RANDCM NUMBER 
C FIX INPUT FIXED EFFECT VECTOR 
C TEMP WORKING MATRIX 
0002 DIMENSION FAC(MJsNE) pERROMJ) pYOMIyMJ) »FIXOMJ), TEMP (MISNF) 
Cc IF(IDISE.EQ.0) CALL SRAND(Y,(MI®MJ), 1X) 
0003 IFC ICISE.EQ.0) CALL GCXSN(Y, (MI®MS D9 IX) 
0004 IF(IDISE.EQe1) CALL DISELY MI yMJyIX) 
0005 CO 20 I=l,MI 
0006 DO 20 J=1,™J 
0007 20 YUL yd) =FIX( SFY (U1, J) FERRIS) 
Cc IF(IDIST.«EQ.0) CALL SRAND( TEMP »(ME®NF),IX) 
0008 LF (TOIST2EQ.0) CALL BOXSN( TEMP y(MI*NF) 5 IX) 
0009 IF(IDISTsEGe1) CALL DIST(TEMP,MI »NFy IX) 
0010 DO 30 I=lyMI 
0011 CO 25 K=l,NF 
0012 CO 23 J=leMJ 
0013 23 YC, J)=¥ (Ty J) +FACO J) KI *TEMPC I,K) 
0014 25 CONTINUE 
0015 30 CONTINUE 
0016 RETURN 
0017 END 


TOTAL MEMCRY REQUIREMENTS OOO50E BYTES 
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FORTRAN IV € CCMPILER EXAMPL 09-15-71 15:356.02 PAGE OOOL 
oool SUBROUTINE EXAMPLOME sMJeNF »FACyERRyIDIST, LDISE, 1X9 Ve FIX, TEMP yBBy 
LFMS,REL) 
PURPOSE GIVES EXAMPLE OUTPUTS FOR RELOL 


0002 


0003 
0004 


0005 


0006 
0007 


0008 


0009 
0010 
ooll 
0012 
0013 
0014 
0015 
0016 
0017 
0018 
0019 
0020 
0021 
0022 
0023 
0024 
0025 
0026 
0027 
0028 
0029 
0030 
0031 
0032 
0033 
0034 
0035 
0036 
0037 
00 38 
0039 


AAA 


ARGUMENTS THE SAME AS THE MAINLINE PROGRAM RELOL 


SUBPROGRAM DATA,MXOUT ,ANOV, VEOUT ,DISP,ROZB 
DIMENSION FAC(MS NF) sERRIMJ) oS Y(MIyMJ Dy FIX( MJ)» TEMPOUMI gNF)»BBIMI) » 


LFMS(MJ_MJ) 


100 FORMAT{ LHL s20(*D") 9/4 591X9°D* 9 3Xe* EXAMPLE RUNS, 3X9°D%y/51X,20( 9" )) 
LOL FORMAT(/91Xy9*MSA=" 5E142632X»— "MSB=" 5E14.692X» *MSE=" E1467 2Xy FH", 
1LE14.652X_* ALPHA=* »F8.592X, "UNBIASED REL EST (ANOVA)=",F8.5) 
FORMAT{(/,1X,"SAMPLE DISPERSION = SATURATION COEFF=" »F9.553Xy *HOMOG 


102 


103 


LENEITY COEFF=" 5F9.5,5X» "HOM/SAT=",F9.5) 
FORMAT (/91X» *GMEAN=* ,E14.6) 


104 FORMAT (/51X,*VARIANCE OF ALPHA ESTIMATE UNDER ANOVA=",F8.55 3X, 


L°ESTIMATE=" ,F9.5) 


105 FORMAT(/91Xs*VARIANCE OF UNBIASED REL ESTIMATES UNDER ANOVA=",F8.5 


20 


90 


1,3X, "ESTIMATE=° ,F8.5) 
wRITE(6y100) 
CALL CATA(MI,MJeNF,FAC, ERR» LOLST sy IDISEsIXs Vs,FIX»s TEMP) 
CALL MXOUT{YsMI yMJeO9l2,L2HDATA MATRIX ) 
CALL ANOV(Y MI »>MJyFMSA,FMSBy FMSE 1 BB) 
FF=FMSA/FMSE 
AL=1.0-1 .0/FF 
ALL=(2.04+(MI-3.0)¥*AL)/(MI-1.0) 
WRITE(6,101) FMSA,FMSB,FMSE,FF yALyALL 
CALL VEOUT(BByMJ,28,28HSAMPLE FIXED EFFECTS VECTOR ) 
CALL DISP(Y»sMI »MJyFMS,BB) 
CALL VEOQUT(BB,MJy20,20HSAMPLE MEANS VECTOR ) 
SUM=0.0 
CO 20 J=1l,MJ 
SUM=SUM+BB (J) 
SUM=SUM/¥J 
WRITE(6,103) SUM 
CALL MXOUT(FEMS »MJ 94590924, 24HS AMPLE DISPERSION MATRIX) 
CALL ROZBIFMS,MJySAT HOM) 
AL PHKA=HOM/ SAT 
WRITE(6,102) SAT,HOM,ALPHA 
IF(MI.LE.5) GO TO 90 


VF=(2.0*%(MI=100) #(MJ#MI—-MJ—200)) /C(ME-5.0) #(MJ-1 0) ®(MI-3.0 ) ##2) 


VARA=VF*®(1-REL ) #¥*2 
VARE=VF* (1-AL ) **2 
WRITE(6,104) VARA, VARE 
C2={MI-3.0)/(MI-1.0) 
VARA=VARA*C2*C2 
VARE=VARE*C2*C2 
WRITE(6,105) VARA, VARE 
RETURN 

ENO 
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0001 


0002 
0003 


0004 
0005 


0006 
0007 
0008 
0009 


0010 
ooll 
0012 
0013 


_ 0015 
0016 


0018 
0019 
0020 


0022 
0023 


0033 
0034 
0035 


aAAAAAgAAAAMAAAAAA 


ee eth POPRIMJ sNF sFAC, ERR» VAR» DI SyREL, ALPHA, TVAR,y EVAR, FMT FIX, 
MEAN) 


PURPOSE PERFORMS BASIC COMPUTATIONS FOR POPULATION PARAMETERS 
. MJ NO OF PART-TESTS 

NF NO OF FACTORS IN TRUE SCORE 

FAC FACTOR LOADING MATRIX 

ERR ERROR STANDARD DEVIATION VECTOR 

VAR OUTPUT VECTOR FOR VARIANCES OF PARTS 

O1S QUTPUT OISPERSION MATRIX OF PART-SCORE VECTOR 

REL OUTPUT POPULATION RELIABILITY 

ALPHA DUTPUT POPULATION RELIABILITY 

TVAR OUTPUT TRUE SCORE VARIANCE OF TEST SCORE 

EVAR OUTFUT ERROR SCORE VARIANCE OF TEST SCORE 

FMT FORMAT FOR INPUT VECTORS AND MATRIX 

FIX OUTPUT MEAN VECTGR FOR PARTS 

GMEAN OUTPUT GENERAL MEAN 


DIMENSION FAC{MJyNF) yERRIMJ) VAR (MJ) p DIS (MJ 9 MJ D9 EMT (20) 9 FIX (MS) 
100 FORMAT{/y1X, "POPULATION PARAMETERS "y/y1Xy*RELIABILITY® 519Xy F905 9/ 
LUX" ALPHA®,25XyF9e55/9 1X9" TRUE SCORE VARIANCE’ 91 1X9614e69/y 1Xy"ERR 
20R SCORE VARIANCE* ,10X,E14.6) 
101 FORMAT(/,1X,*TRUE SCORE DISPERSION: SATURATION COEFF=" »F9.5 ,5X» "HO 
LMOGENEITY COEFF=",F9.5) 
102 FORMAT(1LH1y37{"9") o/91X9'B"y3Xe" INPUT POPULATION PARAMETERS ',5X, 
L'a" y/,1X,37(%")) 
103 FORMAT(/+1X>*GMEAN=" 5£14.6) 
104 FORMAT(20A4) 
105 FORMAT(/9/91Xy*FORMAT FOR THE DATA*,5X,20A4) 
106 FORMAT(/51X,"PART SCORE DISPERSICN: SATURATION COEFF=',F9.555X, "HO 
LMOGENEITY COEF=",F9.5) 
WRITE(6,102) 
READ(5,104) (FMT(1),1=1120) 
WRITE(6,105) (FMT( I) ,1=1,20) 
READ(S,FEMT) (FIX(JS),J=lyMJ) 
READ(SeFMTICERRG J) yJ=Le MD 
CO 10 I=1,4J 
10 REAC(SyFMT) (FAC(1y J) »J=1yNF) 
CALL VEQUT{FEX yMJy12,12HMEANS VECTOR) 
CALL VEOUT(ERR»MJ,32232HERROR STANDARD DEVIATIONS VECTOR) 
CALL MXOUT (FAC »MJyNFy0924,24HFACTOR LOADING MATRIX  } 
TVAR=0+0 
EVAR=0.0 
DO 15 J=l,™J 
VAR{(J)=0.0 
DO 12 I=1yMJ 
DIS(1,J)=0.0 
CO 11 K=1,NF 
LL DIS(UL, J) =DIS( 1) JIFFAC(T 9K) #FAC (J 9K) 
TVAR=TVAR4DIS(I9J) ; 
12 CONTINUE 
VAR(J)=DIS (Jo J) FERRI J) ERR GS) 
15 EVAR=EVARSERR(J)*ERR(S) 
B{DIS,MJ,SAT +H 
ane PV BUT Dia. Huabs On car zBHTRUE SCORE DISPERSION MATRIX) 
WRITE(6,101) SAT#HCM 
REL=TVAR/( TVAR+EVAR) 
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FORTRAN IV ¢ COMPILER POPR 09-15-71 15256.0% 
0036 DO 16 J=1,MJ 

0037 16 DIS(JeJ)=VAR( J) 

0038 CALL ROZB(DIS,MJySATyHOM) 
0039 CALL MXOUT(DIS»MJyMJ»0,20, 20HDISPERSION MATRIX) 
0040 WRITE(6,106) SAT,HCM 

0041 AL PHA=HOM/ SAT 

0042 WRITE(6,100) RELyALPHA, TVARy EVAR 

0043 GMEAN=0.0 

0044 DO 20 J=1,MJ 

0045 20 GMEAN=GMEAN+FIX (J) 

0046 GMEAN=GMEAN/NJ 

0047 COe2b dats MS 

0048 21 FIX(J)=FIX(G}-GMEAN 

0049 WRITE(6,103) GMEAN 

0050 CALL VEQUT(FIXsMJ,1l6,16HFIXED EFFECTS ) 
0051 CALL VEQUT(VARsMJ,20,20HVARIANCES OF PARTS ) 
0052 DO 23 J=1,MJ 

0053 23 FIX(J)=FIX(JS)+GMEAN 

0054 RETURN 

0055 END 
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FORTRAN IV G CCMPILER OIST O9=15=—T1 15:56.10 PAGE 0001 
0001 SUBROUTINE DISTC TEMP »MI NF IX) 
C PURPCSE CREATE STANDARD RANCOM TRUE SCORE MATRIX FOR RELOL 
C TEMP OUTPUT TRUE SCORE MATRIX 
C MI NO OF RCWS OF TEMP 
C NF NO OF COLS OF TEMP 
C IX SEED ODD INTEGER RANDOM NUMBER 
C**#*e THIS EXAMPLE PRODUCES EXPONENTIAL TRUE SCORES 
0002 DIMENSTON TEMP( MI, NF) 
0003 CALL VECRAN(TEMP,(MI*NF) ,IX) 
0004 CO 20 I[=1,MI 
0005 DO 10 J=1,NF 
0006 10 TEMP(I ,J)=-ALOG(TEMP(T,J))-1.0 
0007 20 CONTINUE 
0008 RETURN 
0009 END 


TOTAL MEMCRY REQUIREMENTS O0025A BYTES 
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FORTRAN IV G COMPILER DISE 09-15-71 15:56.11 PAGE 0001 
ooolL SUBRCUTINE DISE(Y,MI »MJ IX) 
C PURPOSE CREATE STANDARD RANCOM ERROR MATRIX Y FOR RELOL 
C ¥ OUTPUT MATRIX 
C : MI NG OF RCWS OF Y 
Cc’ MJ NO OF CQLS OF Y 
Cc , IX SEED ODD INTEGER RANDCM NUMBER 
C##**THIS EXAMPLE PRODUCES UNIFORM ERROR SCORES 
0002 DIMENSICN Y(MI5MJ) ; 
0003 CALL VECRAN(Ys(MI*MJ ), 1X) 
0004 ; SQR=SQRT (12.0) 
0005 CO 20 J=ly¥J 
0006 CO 10 I=lyKI 
0007 10 YCI,J)=0 YC 1,5)-025)*SQR 
0008 °20 CONTINUE © 
0009 RETURN 
0010 : END 
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MAIN 09-15-71 15254.53 PAGE OOOL 


CIVISITON OF EDUCATIONAL RESEARCH SERVICES 
UNIVERSITY OF ALBERTA 


SIO Rt to ik tok tok OO IO ROO ROI RO RO ROR IO ROSCOE HOR Bete tek ake 


PURPOSE 


CARD INPUT: 


REMARK: 


SUBPROGRAMS: 


PROGRAMMER: 


SIMULATE UNIT-OICHOTOMOUS(BINARY) ITEM TEST SCORES 
BASED ON NORMAL OGIVE MODEL TO INVESTIGATE SAMPLING 
OISTRIBUTION OF RELIABILITY ESTIMATES 


1. TITLE (20A4) 

2e PARAMETERS(10159F5.5)2NSAMyMI oMJy IX, IOIST,IDISE, 
IPUNCH,I PLOT, MOCE »LBySIG 
NSAM NO OF SAMPLES SIMULATED 


MI NO OF SUBJECTS IN THE SAMPLE 
MJ NO OF ITEMS 
IX ANY ODD INTEGER TO INITIATE RANDOM NUMBER 


IDOIST OPTION FOR THE DISTRIBUTION OF RANDOM 
EFFECTS (TRUE SCORE) 
O-NORMAL 
1-SPECIFIED BY SUBPROGRAM DIST 
IDI SE OPTION FOR THE DISTRIBUTION OF ERROR 
O-NORMAL 
1-SPECIFIED BY SUBPROGRAM DISE 
IPUNCH OPTION FOR CARD OUT PUT OF FREQUENCIES 
O-NO CARD OUTPUTS 
1-CARD OUTPUT REQUIRED 
IPLOT OPTION FOR PLOTS 
O-NOT REQUIRED 
1-REQUIRED 
MODE OPTION FOR ESTIMATICN FORMULA 
O-ALPHA FORMULA( BIASED) 
1-KRISTCF CORRECT ION(UNBI ASED) 
2-BOTH CF ABOVE 


LB OPTION FOR THE NO OF CLASS INTERVALS FOR 
FREQUENCY CALCULATION»24,36 OR 48,ASSMED 24 
SIG SIGNIFICANCE LEVEL FOR EACH TAIL,ASSUMED 
0.05 
Ze FMT FORMAT FOR THE INPUT VECTORS 
4. DIF A.VECTOR OF ITEM OLFFICULTIES 
5. BS A VECTOR OF BISERIAL CORRELATIONS 


6. A BLANK CARD 


1.CURRENTLY DIMENSIONED TO ACCOMODATE UP TO FOLLOWING 
SIZE PARAMETERS 


NS AM 5000 
MI 100 
MJ 30 
LB 48 


(FORTRAN) ANOV+ BOX SN »CHI PRB, COUNT, DI SCRP yDISPy EXAMPL» 
FISHER sFITTESsFSTs IT EMCO sMXOUT » PARALL y PLOT, POPR y PUNCH 
REL DIS »ROZBy SIGTES sVARXX» VECRANy VEQUT» DATA, DIST yDISE 
(*SSPLIB) BDTR»COTRy OLGAM,NDTRyNDTRI yRANK 


K eBAY 


MAX=LARGER OF NSAM AND MI*MJ 
A teeae can BS (MJ) »ERR(MJ) sVAR(MJ) pDIS (MJ*MS) eDIF (MJ), BSUMIMJ ), 


1LBSS(MJ) 9 YOMI%MJ) 9 BBU MJ) o FMSUNS AM*3 ), FREQ(LB*3),TEMP(MI),RROUMI)» 
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FORTRAN IV G COMPILER PAIN 09-15-71 15254.53 PAGE 0002 
Cc RUM I*(MJS4+19/2),XBAR(2*MJ5), XVAR (MS) 
oool DIMENSION TITLE(20),FMT(20),LAB( 4),BS( 30),ERRE 30),VAR( 30), 


IDIS( 900)-,DIF{ 30) »,BSUM( 30),BSS( 30),Y( 3000),BB( 30),FMS¢ 15000) 
29FREQ(144) yXBAR(60)5XVAR(30),TEMP( 100),RR( 30), PP( 30)+R( 465), 


2X( 3000) 
0002 REAL*8 LAB 
0003 DATA LAB/*SUBJECT*,* ITEMS ", “ERROR*, "REL COF*/ 
0004 100 FORMAT (20A4) 
0005 101 FORMAT (1H1,20A4) 
0006 102 FORMAT(1015,F5.5) 
0007 103 FORMAT(/,10Xs*NO OF SAMPLES SIMULATED® 515X_ 149/510Xy"NO OF SUBJEC 


1TS IN EACH SAMPLE* sLOX913s/910X, "NO OF ITEMS* »28X¥139/,10Xy *STARTI 
2NG SEED RANDOM NUMBER® sy SXeI105/910X,"OPTION FOR CARD OUTPUT',19X, 
3I1ly/,10X,*CPTION FOR PLOT® ,26XyI1y/y10Xy*OPTION FOR ESTIMATICN FOR 
4MULAS S12Xy9T19/510Xs*OPTIION FOR CLASS INTERVALS® 512X,139/,10Xy 
5*SIGNIFICANCE LEVEL* 4519X,F5.3,/) 


0008 104 FORMAT( 2X, 13595X_2E142692X5%1%%2X92E14.6) 
0009 105 FORMAT (/ 1X, ee eee eeL AST SEED RANDOM NUMBER IX=*,I110) 
0010 106 FORMAT(/s1X,*DISCRIPTIVE STATISTICS FOR FIXED EFFECT ESTIMATES AND 


1 EXPECTED VALUES UNDER MeFe MCDEL%s/,1Xy*PART®%s 7Xy*MEAN' LOX," EXP 
ZECTED® 96X_"] *95X9" VARIANCE", 8X," EXPECTED!) 


0011 107 FORMAT(1H1L 230°") pf pLX_e*D%y2X_9* SUMMARY GF QUTPUT® 52X4°Q%9/91Xs 231 
1"a*)) 

0012 L108 FORMAT(1LOX,*ERROR SCORE DISTRIBUTIONS ARE NORMAL *) 

0013 109 FORMAT(10X,*ERROR SCORE DISTRIBUTIONS ARE NOT NORMAL®) 

0014 110 FORMAT(1LOX,*TRUE SCORE OISTRIBUTIONS ARE NORMAL ® ) 

OO15 111 FORMAT(LOX,* TRUE SCORE DISTRIBUTIONS ARE NOT NORMAL ®) 

0016 10 READ(5,100) TITLE 

OO1L7 IF(TITLE(1)2EQ.TITLE(2)) GO TO 99 

0018 WRITE(6,101) TITLE 

0019 REAN(55102) NSAM»MI MJ, IX, IDIST, IDISE, I PUNCH,I PLOT ,MODE,LB, SIG 

0020 IF(SIGLL-LE.O.0) SIG=0.05 

0021 IF(LB.NE.24 -AND.~LBeNE*36-ANDoLBeNE.48) LB=24 

0022 WRITE(6,103) NSAM,MI yMJ oI Xe TPUNCHsI PLOT »MODE,LBy SIG 

0023 TFC IDIST.EC.O) WRITE (6,110) 

0024 IFC IDIST.EG.1) WRITE(6,111) 

0025 IFC ICISEEQ.0) WRITE(6,108) 

0026 [FC IDISE.EQe1) WRITE{6,109) 

0027 CALL POPR(MJ,BS,D0I Fs ReREL», ALPHA, FMT yRRyPP,yERRyTEMP,TVAR,EVAR,DIS) 

0028 IR=0.0 

0029 CO 20 J=1,¥J 

0030 BSUM(J)=0.0 

0031 XVAR{(J)=0.0 

0032 XBAR(J)=0.0 

00 33 JJ=MI+J 

0034 XBAR(JJ}=0.0 

0035 BSS(J)=0.0 

0036 00 19 I[=l,J 

0037 IR=IR+1 

0038 19 R(IR)=0.0 

0039 20 CONTINUE 

0040 CALL VECRAN(Y,100, IX) 


CALL EXAMPLOML » MJy BS pERR TEMPyRRy VoL Xp IDIST yIDISE,X »XBARyBB FMS» 


LREL,RyXVAR) 
0042 DO 50 NTRIAL=1_NSAM 
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FORTRAN IV G COMPILER MAIN 09-15-71 15254.53 PAGE C003 
ne CALL DATA(MIsMJ9BS ,ERRyRRy Vo Xp TEMP» IXy IOIST » LOLS Ey XBAR Re XVAR) 
00 44 CALL ANOV(YsMI9MJ,FMSA,FMSByFMSE , BB) 

0045 FMS(NTRIALJ=FMSA 
0046 [L=NTRIAL+NSAM 
0047 FMS( 11 )=FMSB 
0048 LI=T1+NSAM 
0049 FMS( IT )=FMSE 
0050 CO 45 J=1yMJ 
0051 BSUM{J)=BSUM(J)+BB (J) 

0052 45 BSS(J)=BSS(J)+BB(J)*#2 
0053 50 CONTINUE 

CALL MXOUT(FMSsNSAMs390144,44HMEAN SQUARES: COL~1 MSA,COL-2 MSB,CO 

0054 CALL PARALL(RsXBARs MI yMJyNSAMy XVAR pT VAR2 yEVAR2 gREL2) 
0055 WRITE(6,107) 

0056 IF( 1D1ST.EQ.O.AND. ISISE.EQ.0) GO TO 53 
0057 TVAR=TVAR2 
0058 EVAR=EVAR2 
0059 REL=REL2 
0060 DO 51 J=1,MJ 
0061 51 DIF(J)=XBAR(J) 

0062 53 TVAR=TVAR/ (MJ#MJ) 

0063 EVAR=EVAR/MJ 
0064 THETA=TVAR/EVAR 
0065 DIV=1.0+J*THETA 
0066 GMEAN=0.0 
0067 DO 54 J=1,MJ 
0068 54 GMEAN=DIF(J)+GMEAN 
0069 GMEAN=GMEAN/MJ 
0070 00 55 J=1,J 
oo71 55 DIF(J)=DIF(J)-GMEAN 
0072 NN=NSAM*3 
0073 XMAX=0 20 
0074 00 56 I=1,NN 
0075 56 IF(FMS(1).GT.XMAX) XMAX=FMS(I) 

0076 NN=XMAX/10 2041.0 
0077 XMAX=NN* 10.0 
0078 TINT=XMAX/LB 
0079 CALL CCUNT (EMS NSAMy 390005 TINT gL By FREQ 4X BAR y XVAR) 
0080 XMIN=0.0 
0081 XMAX=TINT#LB 
0082 XBAR (4) =J*TVAR#EVAR 
0083 XXXX=0.0 
0084 DO 58 J=1,¥J 
0085 5B XXXX=XXXX+DIF US) ##2 
0086 DFA=MI-1 
0087 DFB=MJ-1 
0088 DFE=(MI-1) #(MJ-1) 

0089 XXXX=XXXX/OFB 
0090 XBAR (5 )= EVAR #XXXX*MI 
0091 XBAR(6)=EVAR 
0092 XVAR (4)=(20#( MJXTVARFEVAR )**2/DFA 
0093 XVAR(5)=((EVAR+2 eO#MIMXXXX #2 .0*EVAR) /DFB 
0094 XVAR (6) =(2.0*E VAR#EVAR) /OFE 
0095 CALL CISCRP(3,XBARsXVAR,LABL 1) y52252H MEAN SQUARES AND EXPECTED VA 
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0109 
0110 
Olll 
0112 
0113 
OL14 
0115 
0116 


OLLT 


0118 
0119 
0120 
0121 


IV G COMPILER MAIN 09-15-71 


60 
61 


63 


70 


ee) 


CALL VARXX(NSAM,MJ,BSUM, BSS) 
WRITE(6,106) 

CO 63 J=1l,¥J 

XX=0.0 

DO 61 M=1,¥J 
Dl=-1.0/MJ 
IF(JeEQe4) D1=D1+1.0 
DO 60 K=1,J 
D2=-1.0/J 
IF(JeEQ.K) D2=D2+1.0 
MK=N J*¥(M-1) 4K 
XX=XX+D1*D2*0N1S (MK) 
CONTINUE 

XX=XX/MI 


WRITE(65104) J_eBSUM( J) ,DIF (JS) 5 BSS( J) XX 


TI=NSAM*2 

CO 70 I=1,NSAM 

TI=fi+l 

FMS(1)=1.0-FMS(TI) /FMS(T) 

CALL RANK(FMS(1)_,FMS (NSAM#+1) sNSAM) 


15254.53 


188. 


PAGE 0004 


IF(MODE.NE-1L) CALL RELDIS(FMS,NSAM,DFA ,DFEs FREQ, LByRELsTEMPs SIG, 


LXBAR sXVARy, IPUNCH, I PLOT,O,LAB) 


IF (MODE.NE.O) CALL RELDIS(FMSs NSAM,DFA,DFE,FREQ,LByRELy TEMP 9SIGy 


1XBAR,XVARy, IPUNCH, I PLOT 51 » LAB) 
WRITE(6,105) IX 

GO TO 10 

STaP 

ENO 
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FORTRAN [V C COMPILER CATA 09-15-71 15:55.01 PAGE OOO1 


oool Pete DATACMI sMJeBS ERR gRRye VYoXe TEMPS IX ep IDISTyISISE,XBARyR, 
XVAR 
Cc PURPOSE CREATES DATA MATRIX FOR RELO2 
Cc MI SAMPLE SIZE 
Cc MJ NO OF ITEMS 
c BS INPUT VECTOR OF BISERIAL CORRELATIONS 
Cc ERR A VECTOR OF STANDARD DEVIATION OF ERRORS 
C RR A VECTOR OF THRESHOLD CONSTANTS FOR EACH ITEM 
c Y DATA MATRIX 
Cc X WORKING VECTOR 
C TEMP A WORKING VECTOR 
Cc 1X SEED RANDOCM NUMBER 
c IOIST OPTION FOR TRUE SCORE DISTRIBUTION 
Cc IDISE OPTICN FOR ERROR SCORE OISTRIBUTION 
Cc XBAR SUM OF SCORES FOR EACH ITEM 
Cc R INTER ITEM SUM OF PRODUCTS 
Cc XVAR A VECTOR OF PARALLEL ITEM SUM CF PRODUCTS 
c DIMENSION R(MJ*(MJ4#1)/2) »XBAR( 2*MJ) 
0002 CIMENSICN BS(MJ),ERROMJDsRRIMS Do Y(ME oMJS) yXOMEoMJ),TEMP(MID RILD, 
 LXBAR(1),XVAR(MJ) 
0003 MM=MI®MJ 
0004 IF(IOISE) 10,10,12 
0005 “AO CALL BOXSN(YsMMyIX) 
0006 oes CALL BOXSN(XsMMyIX) 
0007 GO TO 15 
0008 12 CALL DISE(YsMI sMJeIX) 
0009 CALL DISE(XsMIyMJy IX) 
0010 15 IF(IDIST) 16516518 
OO11 16 CALL BOXSN(TEMP »MI 91 X) 
0012 GO TO 19 
0013 18 CALL DIST(TEMP,MI, 1X) 
0014 19 CONTINUE 
0015 00 70 I=1,MI 
0016 DO 20 J=ls¥J 
OOLT CUT=RR(J) 
0018 TR=BS(J)*TEMP(T) 
0019 YY=TRHERR(J)¥*Y( 19d) 
0020 Y(1,J)=0.0 
0021 IF(YY.GE-CUT) Y(I,yJ)=1.0 
0022 YY=TRHERR( J) ¥*X(1 2d) 
0023 X(1,J)=0.0 
0024 IF(YY.GE.CUT) X(1,J)=1.0 
0025 20 CONTINUE 
0026 DO 30 J=1,J 
0027 XBAR(J)=XBAR(JSI+Y( 1, J) 
0028 JJ=MJ+J 
0029 30 XBAR(JJ)=XBAR( JS) #X( 1, J) 
0030 IR=0 
0031 DO 50 J=1l,MJ 
0032 00 40 K=1,J 
0033 IR=IR+1 
00 34 40 RUIRJ=ERCIRDFYC LK) ¥V¥ OTe J) 
0035 50 XVAR(J)=XVAR(J) #¥ (19 J) eX TJ) 
0036 70 CONTINUE 


0037 RETURN 
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FORTRAN IV G COMPILER CATA 09-15-71 15:55.01 PAGE 0002 
0038 ENO 


TOTAL MEMCRY REQUIREMENTS 000730 BYTES 
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FORTRAN IV C€ COMPILER EXAMPL 09-15-71 152:55.03 PAGE OOOL 


0001 


0002 


0003 
0004 


0005 


0006 
0007 


0008 


0009 
0010 
OOl1L 
0012 
0013 
0014 
0015 
0016 
0017 
0018 
0019 
0020 
0021 
0022 
0023 
0024 
0925 
0026 
0027 
0028 
0029 
0030 


ANAANANRAAANANAAAAANANANAAMAN 


SUBROUTINE EXAMPLIMI sMJyBS,ERR»yTEMPyRRo Vy IX,IDISTs IDISEsXsXBAR»BBy 
LEMS REL »Ry XVAR) 


MI SAMPLE SIZE 
MJ NO OF ITEMS IN THE TEST 
BS A VECTOR OF GISERIAL CORRELATIONS FOR EACH ITEM 
ERR A VECTOR OF STANDARD DEVIATION OF ERRORS 
TEMP A WORKING VECTOR 
RR A VECTOR OF THRESHOLD CONSTANTS FOR EACH ITEM 
Y DATA MATRIX 
IX SEED RANDCM NUMBER 
IDIST OPTION FOR TRUE SCORE DISTRIBUTION 
IOISE OPTION FOR ERROR SCORE DISTRIBUTION 
X WORKING VECTOR 
XVAR SUM OF ITEM SCORE 
BB WORKING VECTOR 
FMS WORKING MATRIX 
REL POPULATION RELIABILITY 
R WORKING VECTOR 
XVAR SUM OF PRODUCTS OF PARALLEL [TEMS 
SUBPROGRAM CATA,MXCUT » ANOV,VEOUT »OISP,ROZB 


DIMENSION XBAR(2*MJ) yR(MJE(MJ41)/2) 
DIMENSION BS(MJ),ERR(MJ)yTEMPUMI ) yRR(MJ) yY OMI MJ), BB(MJ) » 
LEMS (MJ 9 MJ) 9X (MI 9MJ)y XBAR(L)¢ROL) »XVARCMS) 
100 FORMAT(LHL 20( 9D") y/91Xy!d"%y3Xy* EXAMPLE RUNS") 3X 942% y/y1Xy 20 '")) 
101 FORMAT(/ 1X, *MSA=" 4E14.652Xy "MSB=" 5E140642X 9 "MSE=" 9E14.672X 9 "FH", 
1E14.652X_ "KR20="yF8.5, 2X, "UNBIASED REL EST(ANOVA)=" 4 F8.5) 
102 FORMAT(/y1Xy"SAMPLE DISPERSION 2 SATURATION COEFF=* »F9.5, 3X, 'HOMOG 
LENEITY COEFF=* 9F9.5y 5X, "HOM/SAT=", F925) 
103 FORMAT(/,1X»*GMEAN=" ,E14.6) 
104 FORMAT(/,1Xy*VARIANCE OF ALPHA ESTIMATE UNDER ANOVA=")F8.5,% 3X, 
L*ESTIMATE=',F805) 
105 FORMAT(/,1X,*VARIANCE OF UNBIASED REL ESTIMATES UNDER ANOVA="',F8.5 
1, 3X» "EST IMATE=" »F8.5) 
WRITE (6,100) 
CALL DATA(MI yMJ9BS pERRyRRyYV9Xo TEMP IX» [DIST gy IDIS Ey XBAR y Ry XVAR) 
CALL MXOUT(YsMIyMJ0712,12HDATA MATRIX ) 
CALL ANOV(Y sMI 9 MJ» FMSAy FMS By FMSE y BB) 
FF=FMSA/FMSE ° 
AL=1.0-1.0/FF 
ALL=(2.04+{MI-3 20) #AL )/(MI-1.0) 
WRITE(6,101) FMSAyFMSByFMSE, FF AL, ALL 
CALL VEOUT(BByMJ,28) 28HSAMPLE FIXED EFFECTS VECTOR ) 
CALL DISP(YsMIyMJ9FMSy 8B) 
CALL VEOUT(BB»yMJ»920,20HSAMPLE MEANS VECTOR ? 
SUM=0.0 
DO 20 J=1,¥J 
20 SUM=SUM+BB(J) 
SUM=SUM/ MJ ov 
6,103) SU 
EAU Gites Maei! 0724, 24HSAMPLE DISPERSION MATRIX) 
CALL RCZB(FMSsJySAT yHOM) 
ALPHA=HOM/ SAT 
WRITE(6,102) SAT»HOM,AL PHA 


F(MI.LE.5) GO TO 90 
abe O* (MI -1.0) *(MJ#MI-MJ—-200))/( (MI-5.0) #(MJ-1L 20) *(MI-3 20 2 **2) 
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FORTRAN IV € COMPILER EXAMPL 09-15-71 15:55.03 
0031 VARA=VF*(1-REL) *¥*2 

0032 VARE=VF*(1-AL )**2 

0033 WRITE(69104) VARA, VARE 

0034 C2=(MI-3.0)/(MI-1.0) 

0035 VARA=VARA¥*C2*C2 

0036 VARE=VARE*C2*C2 

0037 WRITE(6,105) VARA, VARE 

0038 90 RETURN 

0039 END 


TOTAL MEMCRY REQUIREMENTS OO0A58 BYTES 
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FORTRAN IV G COMPILER ITEMCO 09-15-71 15355.06 PAGE OOO] 

0001 SUBROUTINE ITEMCO( X,Y,RXY, 01,02, COV) 
C PURPOSE CALCULATE INTER-I TEM COVARIANCE FOR RELO2 BASED ON 
Gc , TCHEBYCHEFF-HERMITE POLINOMIALS UNDER NORMAL OGIVE 
Cc MODEL . 
(e xX THRESHOLD CONST FOR FIRST ITEM 
C Y THRESHOLD CONST FOR SECOND ITEM 
Cc RXY INTER ITEM TETRACHORIC CORRELATION 
Cc 01 NORMAL DENSITY AT Z=X 
Cc D2 NORMAL DENSITY AT Z=Y 
Cc COV OUTPUT COVARIANCE 
C 200 FORMAT(1X,13,E16.8) 

0002 REAL*8 X1¢X2eY1,Y2 eRRR»DDUD aRNeg Us Ve X32Y3,0025FI 

0003 U=X 

0004 V=Y 

0005 X1=1.0 

0006 X2=X 

0007 Y1=1.0 

0008 Y2=Y 

0009 FI=2.0 

001LO RN=OLOG(FI ) 

Ooll RRR=ERXY 

0012 DDD=RRR+ (UFVFRRRERRR)/2.0 

0013 RRR=OLCG(RRR ) 

0014 DO 10 [=3,20 

0015 FI=I 

0016 X3=U*X2—(FI-2-0)*XL 

OOL7 X1=X2 

0018 X2=X3 

0019 Y 3=V*¥Y2-(FI-2.0)*Y1L 

0020 yYl=Y2 

0021 Y2=Y3 

0022 RN=RN+DLOG(IFI) 

0023 DD2=X3#Y3*(NEXPL(FI*RRR-RN) ) 

0024 pDD=000+002 
Cc COV=DD0*D1 #02 

- C WRITE(6,200) I,COV 

0025 10 CONTINUE 

0026 CCV=0D0*D1*D2 

0027 RETURN 

0028 END 
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IV € COMPILER PARALL 09-15-71 15:355.07 PAGE 0001 
SUBROUTINE PARALL(R,XBAR»gMT, MJ »NSAM,XVAR,FC,)FERR,COR) 
URPOSE CALCULATE TEST VARIANCE AND COVARIANCE BY PARALLEL 
METHOD 
R INPUT INTER ITEM SUM OF PRODUCTS 
XBAR INPUT SUM OF ITEM SCORES 
MI SAMPLE SIZE 
MJ NO OF ITEMS IN THE TEST 
NSAM NO OF SIMULATION RUNS 
XVAR SUM OF PRODUCTS GF PARALLEL ITEMS 
FC OUTPUT TRUE SCORE VARIANCE 
FERR OUTPUT ERROR SCORE VARIANCE 
COR OUTPUT RELIABILITY BETWEEN PARALLEL TESTS 


aanananaanaann 


DIMENSION REI MS*(MJ+1)/2) > XBAR(2*MJ) 
DIMENSION R(1),XBAR(1),XVAR(MJ) 


100 FORMAT (LH1] s600°D*)/91X_°d"* »2X_ "ESTIMATION OF POPULATION PARAMETERS 
1 BY PARALLEL METHOD® 42X%5°D%_/_ 1X y60( 8 D"*) gS yLXy*MEAN® pL OXVEL4SeO9/ 

21Xy*VARTANCE® 512X9E140659/51X9* TRUE VARIANCE" »7XyE14.65/91Xy *ERROR 

BVARIANCE "5 6X 9E14-.69/ 91Xe*RELIABILITY® s1LOX9F9.6e/ pl Xy"KR20%y LTXy 


14 


15 


20 


4F9.69/y1X_y "NO OF CASES g11Xy 18) 
NNN=(NSAM#+1)4MTI 

DFN=NNN-1.0 

FC=0.0 

FV=0.0 

FD=0.0 

IR=0 

DO 15 J=1ls"J 

DO 14 K=1l),J 

[IR=1IR+1 

RC IRI=(RCIRI—CXBAR (K) *X BAR (J) /NNN) /DFN 
FV=FV4+R(IR) 

CONTINUE 

FD=FO+R( IR) 

JJ=MI+J 
XVAR(JI=(XVAR(JI—(XBAR( JS) *XBAR (SJ) /NNN))/DEN 
FC=FC+XVAR{(J) 

CONTINUE 

FC=2 .0*(FV-FD) +FC 

FV=2.0*(FV-FD)+FD 
F20=(J*(1.0-FD/FV))/(MJ-1.0) 

FMEAN=0.0 

DO 20 J=ly»¥J 

XBAR (J)=XBAR( J) /NNN 

FMEAN=FMEAN+XBARIJ) 

COR=FC/FV 

FERR=FV-FC 

WRITE(69100) FMEANsFV,FCyF ERR» COR, F20% NNN 
CALL VEOUT(XBAR»MJel2,12HMEAN VECTOR ) 
CALL VEQUT(XVAR)MJ 928s28HPARALLEL ITEM COVARIANCES ) 
CALL MXOUT(R»MJpMJol932e32HWITHIN TEST DISPERSION MATRIX 


RETURN 
END 


TOTAL MEMCRY REQUIREMENTS C©C0692 BYTES 
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IV G COMPILER POPR 09-15-71 153:55.08 PAGE OOO1 


AMDAGAANRANAARANAAARAAAAA 


SUBROUTINE POPR(MJ»BS,DIF,RyRELy AL PHAy FMT, RRyPPy ERR, TEMP, TVAR, 
LEVAR, DIS) 


PURPCSE PERFORMS BASIC COMPUTATIONS FOR RELO2 POPULATICN 
PARAMETERS 

MJ NO OF ITEMS 
BS A VECTOR OF BISERIAL CORRELATIONS 

. OIF - A VECTOR OF ITEM OIFFICULTY 
R INTER ITEM CORRELATION MATRIX 
REL POPULATION RELIABILITY 
ALPHA POPULATION ALPHA COEFFICIENT 
FMT FORMAT FOR INPUT VECTORS 
RR A VECTOR OF THRESHOLD CONSTANTS FOR EACH ITEM 
PP A VECTOR OF ITEM DIFFICULTY ,»REPLACED BY S.De. 
ERR A VECTOR OF STANDARD DEVIATION OF ERRORS 
TEMP A WORKING VECTOR 
TVAR POPULATION TRUE VARIANCE UNDER NeO. MODEL 
EVAR POPULATION ERROR VARIANCE UNDER NO] MODEL 
DIS OUTPUT INTER ITEM DISPERSION MATRIX 


DIMENSION BS(MJ) ,DIF(MJ) »RIMJSe MJ) »FMT(20),RROUMJ) »PP( MS) pERROIMID » 
LTEMP(MJ) sDIS(NJI,MI) 

100 FORMAT{(/,/51Xs*daAd0 =" y13e*TH ITEM DIFFICULTY IS LESS THAN 0.0 OR 
1 GREATER THAN 1-0 DIF=",E&14.5) 

101 FORMAT(/,1Xe9*POPULATION PARAMETERS UNDER NORMAL OGIVE MODEL *) 

102 FORMAT (1X, "MEAN=" ,E14.653Xy* VAR=* 5E614.693Xy*TRUE VAR=",E14-693X, 
L®ERROR VAR="5E14e693X_5"REL=9 9 F 7059 3Xo KRZ20="'5F 7.5) 

103 FORMAT(/1X5"ITEM PARAMETERS o/y1Xe*ITEM%, 2X9 "BIS COR*,10X, *DIFFIC 
LUL TY", 7X» *VARIANCE® »9X—e*®THRES CONS." s6Xe*DISC. POWER® »6X,"DIFF. IN 
20EX*) 

104 FORMAT(1X,1391X96(E14.653X)) 

105 FORMAT (20A4) 

106 FORMAT(/,1X2*FORMAT FOR ITEM PARAMETERS'* ¢5Xy20A4) 

READ(5,105) (FMT(I),1=1 520) 
WRITE(6,106) (FMT(I),1=1,20) 
READ(5,FMT) (DIFC J), J=1l_9MJ) 
READ(5,FMT) (8S( J) eJ=L» MJ) 
DO 10 J=1ly"J 
00 10 [=loJ - 
R(I,JJ=BS(J)*BS(T) 
10 R{J,f=RU0sJ) 
CALL MXOUT(R MJ 9M 90 944e44HINTER ITEM TETRACHORIC CORRELATION MATR 
LIX ) 
WRITE(6,103) 
DO 25 J=l,¥J 
BIS=BS(J5) 
DIFF=DIF(J) 
PP(J)=DIFF*(1-0-DIFF) 
ERR(J)=SQRT(1-0-BIS*BIS) 
AA=BIS/ERR(J) 
CALL NOTRI(DIFF sRRR, TEMP (J), TER) 
IF(IEReNE2O) WRITE(6,100) J,DIFF 
RRR=-RRR 
BB=RRR/(AA*ERR(J)) 
RR(JP=RRR 
25 crannies JeBIS,DIFFyPP( J), RRIJS), AA, BB 


00 31 J=leMJ 
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FORTRAN 


0033 
0034 
0035 
0036 
0037 
0038 
0039 
0040 
0041 
0042 
0043 
0044 
0045 
0046 
0047 
0048 
0049 
0050 
0051 
0052 
0053 
0054 
0055 
0056 
0057 
0058 
0059 
0060 
0061 
0062 
0063 
006% 
0065 
0066 
0067 
0068 
0069 
0070 
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30 
31 


32 


33 


50 
51 


NO 30 I=l,yJ 
RXY=BS(1I)¥*B8S(J) 


CALL PTE MC OU RRCTIARE (I) REN, TEMP (T Je TEMP (J) ,DISC 1, J) ) 


DIS(JsIV=DIS(I, J) © 

CONTINUE 

CONTINUE 

SUM=0.0 

COV=0.0 

S$SS=0.0 

REL=0.0 

DO 33 J=1l,MJ 

DO 32 K=1,J ; 

REL=REL+DI S(K, J) 

SUM=SUM+DIF( JS) 

TEMP(J)=DIS( J, J) 

COV=COV+TEMP( J) 

CIS(J,J)=PPF( J) 

SSS=SSS+PP(J) 

CONTINUE 

TVAR=2 .0*( REL-COV) +SSS 
REL=REL*2-COV 

REL=REL/TVAR 

ALPHA= (¥J*(1.0-SSS/TVAR))/(MJ-1-.0) 
SSS=TVAR¥*REL 

EVAR=TVAR-SSS 

WRITE(6,101) 

WRITE(6,102) SUM, TVAR+SSSyEVAR REL ,ALPHA 
CALL VEOQUT( TEMP syMJ,»28,28HPARALLEL ITEM COVARIANCES 
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) 


CALL MXOQUT (DIS »MJy MJ 909 28, 28HINTER ITEM DISPERSION MATRIX) 


DO 51 J=1l,P¥J 
PP(J)J=SQRT{(PP(J5)) 

00 50 K=l,J 
R(JsKV=DIS(SeK) /( PPC JI*PPIK)) 
R(KsJJ=R( J eK) 

CONTINUE 


CALL MXOUT(Re hoo! 32,32HINTER ITEM CORRELATION MATRIX ) 


RETURN 
END 
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FORTRAN IV G COMPILER O1ST 09-15-71 15:55.13 PAGE 0001 
0001 SUBROUTINE DIST{ TEMP MI,IX) 
C PURPCSE CREATE STANDARD RANDOM TRUE SCORE MATRIX FOR RELO2 
Cc TEMP OUTPUT TRUE SCORE VECTOR 
Cc MI SAMPLE SIZE. LENGTH OF TEMP 
Cc IX SEED ODD INTEGER RANDCM NUMBER 
CHRERTHIS EXAMPLE PRODUCES EXPONENTIAL TRUE OR LATENT SCORES 
0002 DIMENSION TEMP(MI) 
0003 CALL VECRAN( TEMP yMI ys IX) 
0004 CO 10 I=1,MI 
0c05 10 TEMP(T )=-ALCG(TEMP(1))-1.0 
0006 RETURN 
0007 END 
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FORTRAN IV @ COMPILER: DISE 09-15-71 15:55.13 PAGE 0001 
oool SUBROUTINE DISECY,MI »MJ,IX) 
Cc PURPOSE CREATE STANDARD RANOOM ERROR MATRIX Y FOR RELO2 
Cc Y CUTPUT MATRIX 
C Mi NO OF ROWS OF Y 
C MJ NO OF CCLS OF Y 
Cc Ix SEED ODD INTEGER RANDOM NUMBER 
0002 DIMENSION Y(MI,MJ) 
0003 CALL VECRAN(Y,(MI*MJ),IX) 
C SQR=SQRT(1220) ; 
0004 SQR=SQRT (12-0) 
0005 DO 20 J=1,¥J 
0006 DO 10 [=1,MI 
0007 10 Y(I,JI=(Y(155)-0.25)*SQR 
C*¥***THIS EXAMPLE PRODUCES UNIFORM ERROR SCORES 
0008 20 CONTINUE 
0009 RETURN 
0010 ENO 


TOTAL MEMCRY REQUIREMENTS 000264 BYTES 
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FORTRAN IV G COMPILER ANOV 09-15-71 15255.18 PAGE 0001 

0001 SUBROUTINE ANOV(Y»MI 9 ¥JyFMSA,FMS By FMSE, BB) 
C PURPOSE CALCULATE MEAN SQUARES AND PART-TSEST MEANS FOR RELOL 
Elis Y INPUT DATA MATRIX 
C MI SAMPLE SIZE 
c MJ NO OF PARTS ~ 
C FMSA MEAN SQUARES FOR SUBJECT EFFECTS 
C FMSB MEAN SQUARES FOR ITEM EFFECTS 
c FMSE MEAN SQUARES FOR ERRORS 

Kigae2 BB OUTPUT ITEM MEAN VECTOR 

0002 DIMENSION Y(MIgMJ) ,BB(MJ) 

0003 $1=0.0 

9004 52=0.0 

0005 : $3=0.0 

0006 $4=0.0 

0007 DO 15 [=1,MI 

0008 FMSA=0.0 

0009 CO 12 J=lyNJ 

0010 12 FMSA=FMSA+Y(1yJ) 

0011 S4=S4+FMSA 

0012 15 S2=S2+FMSA*#2 

0013 FMSB=S4/ (MI#MJ) 

0014 S2=S2/MJ 

0015 | S4=(S4*S4) /(MI#MJ) 

0016 DO 30 J=1,¥J 

0017 FMSA=0.0 

0018 CO 25 1=1,M1 

0019 SL=S1+V( 1, J) ##2 

0020 25 FMSA=FMSAtY( 1) J) 

0021 BB ( J)=FMSA/MI-FMSB 

0022 S3=S3+FMSA**2 

0023 30 CONTINUE 

0024 $3=S3/MI 

0025 FMSA=(S2-S4)/(MI-1.0) 

0026 FMSB=(S3-S4)/{MJ-1.0) 

0027 FMSE=(S1-S2-S3+S4)/( (MI-1.0)*#(MJ-1.0)) 

0028 RETURN 

0029 END 


TOTAL MEMCRY REQUIREMENTS O004CA BYTES 
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FORTRAN 


oo0ol 


0002 
0003 
0004 
0005 
0006 
0007 
0008 
0009 
0010 
0011 
0012 
0013 
0014 


IV G 


COMPILER BOXSN 09-15-71 15:55.19 
SUBROUTINE BOXSN(ZsNyIX) 

C  puRPOS GENERATE STANCARD RANOCM NORMAL VECTOR 

C z QUTPUT VECTOR OF RANDOM NUMBERS 

Cc N LENGTH OF Z 

c < IX SEED ODC INTEGER RANDOM NUMBER 

C  SUBPROGRAMS VECRAN 

C METHOD BOX-MULLER, ANNs MATH. STAT. 1959 

C DIMENSION Z(2*NN) NN=(N#1L)/2 


20 


DIMENSION 2Z(1) 
PAI=6.283185307T7 
NN=(N4#1) 7/2 

CALL VECRAN(Z,(NN*®2) ,IX) 
00 20 I=l,NN 

XX=PAI*Z(1) 

II=I4NN 

YY=SQRT(-2 OFALOG( ZC 11))9 
ZCL)=VYY*COS (XX) 

ZCI LIY=VVY*SIN(XX) 
CONTINUE 

RETURN 

END 
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FORTRAN IV G COMPILER CHIPRB 09-15-71 15:55. 20 PAGE 0001 
0001 FUNCTION CHIPRB(CHI,NDF) 
C PURPOSE CALCULATE PROBABILITY OF CHI-SCUARE VARIATE EXCEEDING 
C INPUT VALUE 
C CHI INPUT VALUE 
C NOF DEGREES OF FREEDOM 
C PROGRAMMER DO. FLATHMAN 
0002 EXTERNAL ERF,SQRT 
0003 REAL NORMAL 
0004 INTEGER F 
0005 LOGICAL BIGX,EVEN 
0006 NORMAL (X )=0.5*(1.0#ERF(0.7071068*X)) 
0007 F=NOF 
0008 X=CHI 
0009 CHIPRB=1.0 
0010 IF(X.LEs0.OReFeLTol) RETURN 
0011 A=0.5*X 
0012 BIGX=A.GT.10. 
0013 EVEN=(2*(F/2)-F) .EQ.0 
0014 IF(EVEN.OR«(F.GT.2 AND. sNOTeBIGX)) Y=EXP(-A)D 
0015 IF(EVEN) S=¥ 
0016 IF(.NOT.EVEN) S=2.0*NGRMAL (—SQRT (X)) 
0017 CHIPRB=S 
0018 IF(F.LE.2) RETURN 
0019 7 X=0.5*(F-1.0) 
0020 IF(EVEN? Z=1.0 
0021 IF(eNOTSEVEN) Z=0.5 
0022 IF(.NOT.BIGX) GO TO 2 
0023 IF(EVEN) E=0. 
0024 IF(.NOT.EVEN) E£=0.5723649 
0025 C=ALOG(A) 
0026 1 E=ALCG(Z)+E 
0027 S=EXP(C¥Z-A-E) +S 
0028 Z=1+1.0 
0029 IF(Z.LE.X) GO TO 1 
0030 CHIPRB=S 
0031 RETURN 
0032 2 IF(EVEN) E=1.6 
0033 IF( -NOTeEVEN) E=0. 5641896/SQRT(A) 
0034 C=0. 
0035 3 E=E*A/Z 
0036 C=C+E 
0037 Z=Z+1.0 
0038 IF(Z.LE.X) GO TO 3 
0039 CHIPRB=C*Y +S 
0040 RETURN 
0041 END 


TOTAL MEMCRY REQUIREMENTS OO05S5E BYTES 
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FORTRAN IV € COMPILER 


0001 


0002 
0003 
0004 


0005 
0006 
0007 
0008 
0009 
0010 
ool1l 
0012 
0013 
0014 
0015 
0016 
0017 
00 18 
0019 
0020 
oo2l 
0022 
0023 
0024 


0025 
0026 


SUBROUTINE CO 


PURPOSE 


XVAR 
DIMENSION X(L 
XMAX=XMIANtTIN 


COUNT 09-15-71 15:55.23 


UNT(XsLXeNVeXMIN, TINT pL By FREQy» XBARy XVAR) © 


CALCULATE FREQUENCY OISTRIBUTIONS 
INPUT DATA MATRIX 

NO OF OBSERVATIONS 

NO OF VARIABLES 

INPUT MINIMUM VALUE ASSUMED 

INPUT CLASS INTERVAL 

INPUT NO OF CLASS INTERVALS 
OUTPUT FREQUENCY OISTRIBUTIONS 
OUTPUT MEAN VECTOR 

OUTPUT VARIACE VECTOR 

X»NV) sFREQ(LB sNV) yXBAR(NV) »XVAR(NV) 
T*Le 


203". 


PAGE OOO1 


~ 100 FORMAT (/,1Xs"MAXIMUM=" E14.643X_ © MINUMUM=* 4£14.653Xy "CLASS INTERVA 


10 


15 


68 


80 


1L=',E14.6) 
WRITE(6,100) 
DO 15 J=1,NV 
CO 10 [f=1,L8 
FREQ(IsJ)=020 
XBAR(J)=0.0 
XVAR(J)=0.0 
DQ 80 J=1l,NV 
DO 70 I=1l,LX 
XBAR(J)=XBAR ( 
XVAR (J) =XVAR ( 
XXL=XMIN 

DO 60 K=1y,LB 
XXU=XXL+TINT 
IF(X(1e9J)-GE.~ 
XXL=XXU 

GO TO 70 
FREQ(K,J)=FRE 
CONT INUE 
CONTINUE 

CALL VARXX(LX 


CALL MXOUT (FREQ sLByNV20224224HFREQUENCY DISTRIBUTIONS ) 


RETURN 
ENO 


XMAXsXMIN, TINT 


JV#X(1, J) 
JdV4#XC1, JS) #X(T J) 


XXL eAND X(T pS) oLTeXXU) GO TO 68 


Q(Ky,J)+1.0 


»>NVyXBAR,XVAR) 
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FORTRAN IV G COMPILER CISCRP 09-15-71 15255224 PAGE 0001 
oooL SUBROUTINE DISCRP( Ny XBARyXVAR,LAB,y NUMHOL yTITLE) 
Cc PURPOSE OUTPUT DISCRIPTIVE TABLE 
Cc N NO OF VARIABLES 
c XBAR MEAN VECTORS 
c XVAR VARIANCES 
Cc LAB LABELS 
Cc NUMHOL NO OF CHARACTERS IN TITLE( MULTIPLE OF 4) 
Cc TITLE TITLE OF THE TABLE 
0002 100 FORMAT(1HO,*DESCRIPTIVE STATISTICS FOR *,20A4) 
0003 101 FORMAT (LHO y20X 9 *MEAN® 15X59] % yp LOX, VARIANCE 9/91 Xe 11Xe "OBSERVED', 
17Xy *EXPECTED', 5X5" 1" 95X_ "OBSERVED y8X_y "EXPECTED! ) 
0004 102 FORMAT(1LXy ABs, 1X%92E140652X9"] "9 3X 261426) 
0005 DIMENSION XBARIN) »XVARON) »LABCN) sTITLEL20) 
0006 REAL*8 LAB. 
0007 NN=(NUMHCL43)/4 
0008 WRITE(6,10C) (TITLE( J) eJ=1 NN) 
0009 WRITE(6,101) 
O0OLO DO 10 I=1,N 
0011 II=I4N 
0012 WRITE(6,102) LAB(I), XBAR(I),XBARC(IT) »XVAR( I) ,XVARCIT) 
0013 10 CCNTINUE 
0014 RETURN 
0015 END 
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205. 


PAGE OOOL 
oool SUBROUTINE DISP(YsMI »MJ9S,» XBAR) 
Cc PURPOSE CALCULATE SAMPLE CISPERSION MATRIX AND MEAN VECTOR 
Cc Y INPUT DATA MATRIX 
C MI NO OF RCWS OF Y 
Cc MJ NO OF CCLS OF Y 
Cc Ss OUTPUT SAMPLE DISPERSION MATRIX 
C XBAR SAMPLE MEAN VECTOR 
0002 DIMENSION YO(MEyMJ) sS (MJ oMJ) eXBAR (MJ) 
0003 DO 15 J=lyMJ 
0004 DO 10 K=1,¥J 
0005 10 S{K,J)=0.0 
0006 15 XBAR(J)=0.0 
0007 DO 30 I[=1,¥I 
0008 00 25 J=ly¥J 
0009 CO 20 K=l,J 
0010 20 SK, JI=S(K pJI+V IT KD FV(T J) 
ooll 25 XBAR(JI=XBARC J) #V(1, J) 
0012 30 CONTINUE 
0013 0G 50 J=1,¥J 
0014 DO 45 K=1,J 
0015 S(KyJ)=(S (Ks J) —(XBAR (K) *XBAR(J)) /MI)/0MI-1.0) 
0016 45 S(JoK)=S(K oJ) 
0017 50 CONTINUE 
0018 DG 60 J=1,MJ 
0019 60 XBAR{J)=XBAR(J)/MI 
0020 RETURN 
0021 END 
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FORTRAN IV G COMPILER FISHER 09-15-71 15:55.28 PAGE OOO] 


0001 


0002 


0003 
0004 


0005 
0006 


0007 
0008 
0009 
0010 
0011 
0012 
0013 
0014 
OOL5 
0016 
0017 
0018 
0019 
0020 


FUNCTION FISHER(DFM,DFN,FR) 
PURPOSE CALCULATE PROBABILITY LEVEL WITH GIVEN D.F. AND F-RATIO 


DFM INPUT NUMERATOR O.F. 
DFN INPUT DENOMINATOR 0.F. 
FR INPUT F-RATIO 


100 FORMAT(/s/51HO,*ERROR IN FUNCTION FISHERS AN INPUT PARAMETER IS INV 
LALID*) 

101 FORMAT(1LX,*INPUT F-RATIG IS LESS THAN 0.0 F=*,E16.8) 

102 FORMAT(1X» *NUMERATOR D.F. ITS LESS THAN 1.0 CR GREATER THAN 200,000 
1 DF1l=',E16.8) 

103 FORMAT (1X,*DENOMINATOR DoFeIS LESS THAN 1-0 OR GREATER THAN 200,00 
1 DF2=",E£16.8) 


- 104 FORMAT(LHO,"ERROR OUTPUT PROBABILITY IS INVALID OUE TO COMPUTATION 


LAL DIFFICULTY’ s/,1X, *PROBABILITY IS SET AS TO -1.0E75") 
105 FORMAT(1HO,*ERROR IN CALULATING GAMMA FUNCTION® ) 

A=DFR/2.0 

B=DFN/2.0 

FB=((DFM*FR)/DFN)/(1.0+(DFM*FR)/DFN) 

CALL BDTR(FB:AsBePRO5D, IER) 

IF(TER~EQ.-2) WRITE(6,100) 

IF(FReLT.0.0) WRITE(6,101) FR 
IF(DFM.LT.21.0-0R-DFM.~GT~200000) WRITE(6,102) DFM 
IF (DFN.LT.1-0.0ReDFN-oGT~.200000) WRITE(6,103) DOFN 
IF( 1TER.EQ.2) WRITE (6,104) 
IF(ITERJEQ.-1 OR TERS EQe1) WRITE( 6,105) 
FISHER=1.0-PRO 

RETURN 

END 
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FORTRAN [V G COMPILER FITTES 09-15-71 15:55.30 PAGE OOOl 

0001 _ SUBROUTINE FITTES(XsN,NT) 
C PURPOSE PERFORMS CHI-SQUARE GOODNESS OF FIT TEST 
Cc 4 INPUT FREQUNCY MATRIX 
Cc COL-1 EXPECTED FREQUENCIES 
€ COL-2 OBSERVED FREQUENCIES 
Cc N NO OF CLASS INTERVALS OR NO OF THE ROW OF X 
Cc NSAN SAMPLE SIZE 
c SUBPROGRAM CHIPRB 

0002 CIMENSION X(Ny2) 

0003 100 FORMAT(/,1Xe*CHI-SQ GOODNESS OF FIT TEST: "» 3X, "CHI=" 614.65 3X» "NOF 

l=", 15,3X»y" PROB=',F 8.4) 

0004 CHI=0.0 

0005 NDF=0 _ 

0006 XT=0.0 

0007 YT=0.0 

0008 t=1 

0009 12 XX=0.0 

0010 YY=0.0 

0011 15 YY=YY4+X(1I,2) 

0012 XX=XX+#X( 1,1) 

0013 IF(XX.GE.5.0) GO TO 16 

0014 I=I+1 
Cc IF(I.LE.N) GO TO 15 

0015 GO TO 15 

0016 16 NDF=NDF41 

0017 CHI=CHI+{( (XX-YVY )#*2) /XX 

0018 XT=XT#XX 

0019 YT=YT+YVY 

0020 XR=NT-XT 

0021 I=14l 

0022 YR=NT-YT 
Cc IF (XR.GE~1L0-.0-AND-I.LE.N) GO TO 12 

0023 IF (XR.GE.10.0) GO TO 12 

0024 IF (XR.GT~.0.0) GO TO 20 

0025 NDF=NOF-1 

0026 GO TO 25 

0027 20 CHI=CHI+(( XR-YR )**2)/XR 

0028 25 PRO=CHIPRBICHI »NDF) 

0029 WRITE(6,10C) CHI,NOF,PRO 

0030 RETURN 

0031 END 
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FORTRAN IV CG COMPILER FST 09-15-71 15:55.31 PAGE OOOL 
oool FUNCTION FST(OF1,0F2_Py_,PRE) 
Cc PURPOSE CALCULATE F STATISTICS WHEN DEGREES OF FREEDOM AND 
Cc PROBABILITY ARE GIVEN 
Cc DFl DEGREES OF FREEDOM FOR NUMERATOR 
Cc OF2 DEGREES OF FREEDOM FOR DENOMINATOR 
Cc P PROBABILITY LEVEL 
Cc PRE PRECISION LEVEL FOR OUTPUT F RATIO 
C SUBPROGRAM FISHER 
0002 IF (DF1.LE.0.0.OR-DF2.LE.0-0-0R.P-LE.0.0) GG TO 999 
0003 100 FORMAT {(1HO,"*DEGREES OF FREEDOM OR PROBABILITY IS LESS OR EQUAL 
1TO ZERO RETURNS TO MAIN WITH FST=0.0") 
0004 X1=1.0 
0005 X2=0.0 
0006 10 F=(X14#X2)/2.0 
0007 FR=DF2¥*((1.0-F)/(DF1L*F)) 
0008 PRO=FISHER(OF1L,OF2,FR) 
0009 ER=P-PRO 
C 101 FORMAT (1X, 4F12.6) 
C WRITE(6,101) FeFR,PRCVER 
0010 IF(ABS(ER) .LE.PRE) GO TO 99 
ooll IF (PeLT.~PRO) X1=F 
0012 IF(P.GT.PRC) X2=F 
0013 GO TO 10 
0014 99 FST=FR 
0015 RETURN 
0016 999 WRITE( 6,100) 
OOL7 FST=0.0 
0018 RETURN 
0019 END 


TOTAL MEMCRY REQUIREMENTS 000330 BYTES 
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FORTRAN IV G COMPILER MXOUT 09-15-71 15255.35 
oool SUBROUTINE MXOUT(AsNoM~pMS»NUMHCL ,TITLE) 
‘e PURPOSE QUTPUTS A MATRIX 
€ A INPUT MATRIX 
rie N NO OF ROWS IN A 
C M NO OF COLS INA 
(7 MS OPTION FOR STORAGE MODE 
Cc O GENERAL 
Cc 1 SYMETRIC 
Cc 2 DIAGONAL 
c NUMHOL NO OF CHARACTERS OF TITLE(MULTIPLE OF 4) 
C TITLE TITLE OF THE VECTGR IN A FORMAT 
0002 DIMENSION A(1),8(8),TITLE(20) 
0003 100 FORMAT(1HO,20A4) 
0004 ~ LOL FORMAT (/ 95X%e8(5X%sA291396X)) 
0005 102 FORMAT (LX, *R-" 91 3,E1 5269 761606) 
0006 CATA COL/*C-'"/ 
0007 NN=( NUMHOL4+39/4 
0008 WRITE(65100) (TITLE( J) »J=1l yNN) 
0009 LINS=N+4+2 
0010 J=1 
OOl1l LEND=N 
0012 NEND=8 
0013 10 LSTRT=1 
0014 20 CONTINUE 
0015 JNT=J+NEND-1 
0016 IF(JNT-GToM) JNT=M 
0017 WRITE (6,101) (( COL, JCUR) »JCUR=J »JNT) 
0018 LTEND=LSTRT+LEND-1 | 
0019 DO 80 L=LSTRT»LTEND 
0020 DO 55 K=1l,NEND 
0021 KK=K 
0022 JT=J+K-1 
0023 IF (MS-1) 41942945 
0024 41 IRX=N*(JT-L)4L 
0025 GO TO 47 
0026 42 IF(L-JT)43944, 44 
0027 43 IRX=Lt(JT*JT-JTI/2 
0028 GO TO.47 
0029 44 IRX=JIT+(L¥L-L)/2 
0030 GO TO 47 
0031 45 IRX=0 
0032 IF(L-JT) 47946947 
0033 46 IRX=L 
0034 47 IJNT=IRX 
0035 B(K)=0.0 
0036 IFC IJNT) 50550549 
0037 49 BI(K)=A(IJNT) 
0038 50 CONTINUE 
0039 IF( JT-M) 55,60,60 
0040 55 CONTINUE 
0041 ~ 60 WRITE(6,102) L,(B( JW), JW=1,KK) 
0042 IF(N-L) 85,85,80 
0043 80 CONTINUE 
0044 LSTRT=LSTRT+LENO 
0045 GO FO 20 
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FORTRAN IV © COMPILER MXOUT 09-15-71 15255.35 
0046 85 IF(JT-M) 9C,95495 

0047 90 J=JT+1 

0048 GO TO 10 

0049 95 RETURN 

0050 END 


TOTAL MEMCRY REQUIREMENTS OOO5A0 BYTES 
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FORTRAN IV © COMPILER PLOT 09-15-71 15:55.37 PAGE 0001 
0001 SUBROUTINE PLOT(XyAyN)NSAM) 
PURPOSE GIVES PLGT FOR RELOL c 
x INPUT MATRIX OF RELATIVE FREQUENCIES(2) 


0002 
0003 
0004 


0005 
0006 
0007 
0008 
0009 
0010 


ooll 
0012 
0013 
0014 
0015 
0016 
0017 
0018 
0019 
0020 
0021 
0022 
0023 
0024 
0025 
0026 
0027 
0028 
0029 
0030 
0031 
0032 
0033 
0034 
0035 
0036 
0037 
0038 
0039 
0040 


0041 . 


0042 
0043 
0044 
0045 
0046 


AAAAAANH. 


Z COL-1 EXPECTED 
COL-2 OBSERVED 


A WORKING VECTOR 
N NO OF CLASSES(LIMITED TO 24,36 OR 48) 
NSAM SAMPLE SIZE 


DIMENSION X(N,2),A(N),SIG(4) 
DATA SIG/% 2% ,8et, tet ye ty 

100 FORMAT (1H1,34(°2*)5/,1X,*% PLOT OF FREQUENCY DISTRIBUTION %*y5/,1Xy 
134(°%*")) 


_ LOL FORMAT (4X, "FREQUENCY (2) *) 


102 FORMAT(1X,F7.3,°%—-]*%,24(? *,Al,® )) 
103 FORMATI9OX,*]%,24(* "sAle® ")) 
104 FORMAT (3X,y *REL*53X_5* f* ,24(8 1°)) 
L105 FORMAT (4X, 12(F701,3XI,F7o1) 
106 FORMAT(/,1Xs"NOTEZ" 3X" (o) EXPECTED FREQ; (*) OBSERVED FREQ; (+) 
LOVERLAPPING POINT® ) 
107 FORMAT(LX,FT7.39 "4%, 360" "sAly® *)) 
108 FORMAT(9X,*]%,36* %,AL,*® *)) 
109 FORMAT(3X,y "REL" 5 3X9* J] * »36( *--] ")) 
L110 FORMAT (4X,13¢(F7e1,2X)) 
LLL FORMAT(LX,F7e39*—-]*»48(Als* *)) 
112 FORMAT(9X,"]",48(Aly*® *)) 
L113 FORMAT(3X» *REL* 9 3X5°]*%548(*-]*)) 
114 FORMAT(4X,12(F7eLs1X)eF 7.1) 
NNN\=N-36 
wRITE(6,100) 
CALL FITTES(XsyN ,NSAM) 
WRITE(6,101) 
XMAX=0.0 
DO 10 I=1,N 
CO 10 J=1,2 
X( 0, JS)=(X( 1,5) *100.0)/NSAM 
IFOCXC1,5) GT eXMAX) XMAX=X(IeJ) 

10 CONTINUE 
NN=XMAX/ 10 e0+1.0 
XU=NN*10.0 
IF (XU.GT.100.0) XU=100.0 
DELT=XU/50.0 
DELTH=DELT/2.0 
XU=XU+DELTH 
DO 50 I=1,10 
XM=XU-DELTH 
00 40 J=1,5 
XL=XU-DELT 
DO 20 K=1;N 
A(K)=SIG(4) 

LF(X€ Kyl) -GEeXUeORe Xt Kyl)oLT-XL? GO TO 12 
A(K)=SIG(1) 

12 IFCX€ Ky2)-GE.XU-OR-X( Ky2)-eLT-XL) GO TO 20 
IF(A(K)EQ-SI1G(1)IGO TO 14 
A(K)=S1G(2) 

GO TO 20 
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FORTRAN IV G COMPILER PLOT 09-15-71 
0047 14 A(K)=S1G(3) 

0043 20 CONTINUE 

0049 XU=XL 

0050 IF (NNN) 254530,35 

0051 25 IF(JeEQeL) WRITE(6,102) XMy(ACK) pK=19N) 
0052 IF(J.GT.1) WRITE(6,103) (A(K),K=1,N) 
0053 GO TO 40 

0054 30 IF(J.EQe1) WRITE(6,107) XMyC(ACK) sK=19N) 
0055 IF(JS-GTo1) WRITE(6,108) (A(K)yK=1yN) 
0056 GO TO 40 

0057 35 IF(J.EQ.1) WRITE(6,111) XM (ACK) @K=19N) 
0058 IF(JoGT.1L) WRITE(6,112) (A(K),K=1yN) 
0059 40 CONTINUE 

0060 50 CONTINUE 

0061 A(1)=-0.2 

0062 DELT=0.1 

0063 CO 51 1=2,13 

0064 51 ACI)=ACI-1)4DELT 

0065 IF(NNN) 52554756 

0066 52 WRITE(6,104) 

0067 WRITE(64105) (AC 1), 1=1,13) 

0068 GO TO 58 

0069 54 WRITE(6,109) 

0070 WRITE(6,110) (ACI) 5I=1513) 

0071 GO TO 58 

0072 56 WRITE(6,113)-. 

0073 WRITE(69114) (A(T), 1=1913) 

0074 58 CONTINUE 

0075 WRITE( 6,106) 

0076 RETURN 

0077 END 
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FORTRAN IV C€ CCMPILER PUNCH 09-1 5—7.1 
0001 SUBROUTINE PUNCH(FREC,LB,NV) 
Cc PURPCSE GIVES CARN OUTPUT FOR RELOL 
C LB NO OF ROWS OF FREQ 
A C NV NO OF CCLS OF FREQ 
0002 DIMENSION FREQ{(LB,NV) 
0003 100 FORMAT (12,3Xs5F10.4) 
0004 DO 20 I=1,LB 
0005 20 WRITE(7,10C) Ty (FREQ(I 9 J)» J=1lyNV) 
0006 RETURN 
ooCcT ENO 
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REQUIREMENTS 000200 BYTES 
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RELOIS 09-15-71 15255.42 PAGE OOOL 


SUBROUTINE RELDI SC FMS,NSAM DFA pDFE, FREQ, LB, REL» TEMP ySIGeXBARsXVAR, 


1LIPUNCH,IPLOT, 


TD,LAB) 


CIMENSTON FMS(NSAM,2),FREQ(LB,2),TEMP{ 130),XBAR(2),XVAR(2) »LAB(4) 


PURPOSE 


FMS 


XBAR, XVAR 
TPUNCH 
[PLOT 

ID 


LAB 
REAL*8 LAB 


INVESTIGATE SAMPLING DISTRIBUTION OF RELIABILITY 
ESTIMATES 

AN INPUT MATRX 

COL-1 RELIABILITY ESTIMATES 

COL-2 RANK OF ABOVE 

SAMPLE SIZE 

DEGREES OF FREEDOM OF NUMERRATOR 

DEGREES CF FREEDOM FOR ERRORS 

FREQUENCY TABLE 

NO OF CLASS INTERVALS, NO OF RCWS OF FREQ 
POPULATION RELIABILITY 

WORKING VECTOR 

SIGNIFICANCE LEVEL FOR EACH TAIL 

WORKING VECTORS 

OPTION FOR CARD OUTPUT 

OPTION FOR PLOT 

OPTION FOR ESTIMATION FORMULA 

O-BIASED ALPHA FORMULA 

1-KRISTOF CORRECTION, UNBIASED 

LABELS 


100 FORMAT(LH1L 260 °°) 9/,1Xe'D*e2Xy* RELIABILITY STUDY’ 55X5'D%* 9/9 1Xy 


126('d")) 


101 FORMAT(/,1X,*EXPECTED FREQUENCY OF RELIABILITY ESTIMATES BELOW -0. 


12=",E14-6) 


102 FORMAT(/ 51 X,"ESTIMATION IS BASED ON ALPHA FORMULA(ANOVA,BIASED)®) 
103 FORMAT(/,1Xy*ESTIMATION IS BASED ON KRISTOF CORRECTION(ANOVA,UNBIA 


LSEO)') 
WRITE(6,100) 


IF(IC.EQ.0) GO TO 22 


WRITE(6,103) 
Ci=2.0/0FA 


C2=(D0FA-2.0)/DFA 
CO 21° I=1l»yNSAM 
21 FMS(1,1)=C1#+C2*FMS(I51) 


GO TO 23 

22 WRITE(6,102) 
C1=0.0 
C2=1.0 

23 CONTINUE 


FF=(1.0-REL)/1.2 
PL=FISHER( CFA, DFE, FF) 
FF=NSAM*(1.0-PL) 


WRITE( 6,101) 
RR=-0.2 
DELT=1.2/LB 
0O 25 f=2,LB 
RR=RR+DELT . 


FF 


FF=((1-0-REL)#C2)/(1-0-RR) 
PU=F ISHER( CFA, DFE, FF) 
FREQ( (1-1) 91) =NSAM*{ PL-PU) 


25 PL=PU 
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FORTRAN IV € COMPILER RELOIS 09-15-71 15:55.42 PAGE 0002 
0032 FREQ(LB, 1)=NSAM*PL 

0033 CALL CCUNT(FMS(1L91l)eNSAMo1l 5-002, DELTsL By FREQ(L 92)» XBARyXVAR) 
0034 XBAR(2)=REL 

0035 IF(ID.EQ.0) XBAR(2)=-2.0/(DFA-2.0) +(DFA*REL)/( OF A-2.0) 

0036 XVAR (2)=((1.0-REL) #*2) *2.0* (DF A**2) *(DFEtDFA-2.0) 

0037 XVAR (2) =(C2*C2*XVAR{ 2) ) /(DFE* ( OF A-4 20) *( DFA-2.0) #¥2) 

0038 CALL DISCRP(1,XBAR,XVAR,LAB(4) 924,24H RELIABILITY ESTIMATES ) 
0039 CALL MXOUT(FREQsLB 220523, 28HCCMPARISON OF RELIABILITY ) 
0040 IF ( TPUNCK.EQe1) CALL PUNCH(FREQs)LB 2) 

0041 CALL SIGTES(FMS NS AM,OFA,DFEsS IGsREL, ID) 

0042 TF( IPLOT.LEQ.1) CALL FLOT( FREQ, TEMP,LByNSAM) 

0043 RETURN 

0044 END 


TOTAL MEMCRY REQUIREMENTS OCC898 BYTES 


000 39A49 a Sd .eaees 


{AAVKs AARK atSed ELS Bas Td ; 


(Oo SA 20 INE 13RSAAG): 
(0, SAAR 

(S#% (0. 5A INI#LOL? 

( 2ATAMIT29 ¥YTPITBARSIA HAS 
So ¢) O VRLDEBATIGA Aa eae 


4945 tees & 


oe a = 


216. 


FORTRAN IV € COMPILER ROZB 09-15-71 15355.46 PAGE 0001 
0O0OoL SUBROUTINE ROZB(DIS,MJ,SAT,HOM) 
C PURPOSE CALCULATE SATURATION AND HOMOGENEITY COEFFICIETS 
Cc O1S INPUT DISPERSION MATRIX 
C MJ SIZE OF OIS 
Cc SAT OUTPUT SATURATION COEFFICIENT 
C HOM OUTPUT HOMOGENEITY COEFFICIENT 
0002 DIMENSION DIS(MJ_MJ) 
0003 TEMP=0.0 
0004 HOM=0.0 
0005 0O 20 J=1l»¥J 
0006 OO 10 K=1,MJ f 
0007 10 TEMP=TEMP#DIS(KsJ) 
0008 20 HCM=HOMtDIS( JS, J) 
0009 SAT=TEMP/(ROM*MJ) 
0010 HOM=(TEMP-HOM) / (HOM* (MJ-1.0)) 
ooll RETURN 
0012 END. 


TOTAL MEMCRY REQUIREMENTS 000288 BYTES 
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FORTRAN IV G COMPILER SIGTES 09-15-71 153:55.47 PAGE OOO] 
oool SUBROUTINE SIGTES(FMS,NSAM,DFA,OFE sSIGsREL, ID) 
PURPOSE OBTAIN EMPRICAL CRITICAL POINTS OF RELIABILIRY 
é ESTIMATES. 
FMS AN INPUT MATRX 


0002 
0003 


0004 


0005 


0006 
0007 
0008 
0009 
0010 
oo11 
0012 
0013 
0014 
0015 
0016 
0017 
0018 
0019 
0020 
0021 
0022 
0023 
0024 
0025 
0026 
0027 
0028 
0029 
0030 
0031 
0032 
0033 
00 34 
0035 
0036 
0037 


aanaananaaaaaannac 


COL-1 RELIABILITY ESTIMATES 
COL-2 RANK OF ABOVE 


NSAM SAMPLE SIZE 

OFA DEGREES OF FREEDOM OF NUMERRATOR 
DFE DEGREES OF FREEDOM FOR ERRORS 

SIG SIGNIFICANCE LEVEL FOR EACH TAIL 
1D OPTION FOR ESTIMATION FORMULA 


O-ALPHA COEFF ICTENT 
1-KRISTOF CORRECTION FOR BIASEDNESS 
DIMENSION FMS(NSAM 2) 


101 FORMAT(1X, "ALPHA ESTIMATES", 3X, "SIGLEVEL (EACH) =* 5F5.3593X_ *DFA=", 
1F6 .053X» *DFE=" F820, 16X,*"LOWER BOUND=",F9.694Xe*UPPER BOUND=%, 


102 FORMAT(1X,*NO OF CASES LESS THAN LOWER B=* ¢1492XyF6.2,'% 


2F9.6) 


+ GREATER 


1 THAN UPPER B=" 914 92X9F6029°%%y2X_y "LOWER BUEST J=",F9.653Xy"* UPPER B 


1(EST)=",F9.6) 


103 FORMAT(1X,*"ADJUSTEC ALPHA ESTIMATES* ,3X,"SIG LEVEL( EACH)=",F5.3, 
13X99 *DFA="5F6.053X» "DFE=" F820, 6Xy*"LOWER BOUND=" 9F9.6,4X, "UPPER BO 


10 


20 


2CUND=",F9.6) 
C2=(OFA-2.0)/DFA 
IF(IO0.EQ.0) C2=1.0 
FU=FST( DFE sDFAsSIG,0-0001) 
FL=FST(CFA,DFE,SIG,0.0001) 
FL=L.0/FL 
BL=1.0-FU*(1.0-REL )*C2 
BU=1.0-FL¥(1.0-REL)*C2 
IF(IDeEQ.0) WRITE(6,101) SIG,DFA,OFE,BL, BU 
IF(IDeEQe1) WRITE(6,103) SIG,DFA,OFE,BL,BU 
ML=0 
MU=0 
CO 10 I=1l,NSAM 
IF(FMS(I,1)-GT.B8U) MU=MU4+1 
IF (FMS(I,13eLT.BL) ML=ML41 
EML=(ML*100.0) /NSAM 
EMU=(MU*100.0) /NSAM 
NL=NSAM*SIG+1. 50001 
NU=NSAM* (1.0-SIG)+0. 50001 
NLL=NL-1 
NUU=NU+41 
FNL=0.0 
FNLL=0.0 
FNU=0.0 
FNUU=020 
DO 20 I=lyNSAM 
NID=FMS( 1, 2)4+0.500001 
IF(NIDeEGeNL) FNL=FMS(T, 1) 
IF(NIDeEGeNLL) FNUL=FMS(T,1) 
IF(NIDeEGSNUU) FNUU=FMS( I 1) 
IF (NID/ECGeNU) FNU=FMS(T,1) 
SU=( FNU#FNUU)/2.0 
SL=(FNL4FNLLI/2.0 
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FORTRAN IV € COMPILER SIGTES 09-15-71 
0038 WRITE(6,102) ML »EMLs MUyEMUy SL, SU 
0039 RETURN 
0040 END 
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FORTRAN IV G COMPILER VARXX 09-15-71 15355.49 PAGE OOOL 
oool SUBROUTINE VARXX(NeNV_XBAR y XVAR) 
C PURPOSE CALUCULATE MEANS AND VARIANCE VECTORS 
Ce N SAMPLE SIZE 
C NV NO OF VARIABLES 
C XBAR INPUT SUM OF VARIABLES, REPLACED BY MEANS 
C XVAR INPUT SUM OF SQUARES, REPLACED BY VARIANCES 
0002 DIMENSICN XBAR(NV) »XVAR(NV) 
0003 CO 10 J=1lyNV 
0004 XVAR (J3)=(XVAR( J I-CXBAR CS) *®XBAR (SI) /NI/ON-1.20) 
0005 10 XBAR(J)=XBAR(J)I/N 
0006 RETURN 
0007 END 
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FORTRAN IV G COMPILER VECRAN 09515271 15355.50 PAGE OOOL 
0001 SUBROUTINE VECRAN(Z,N, 1X) 
C PURPOSE COMPUTES N UNIFORM RANOOM NUMBERS BETWEEN 0.0 AND 1.0 
C USING SSP RANDU METHOD 
-C Z OUTPUT RANDCM VECTOR 
Cc N LENGTH OF Z 
Cc. Ix SEED OND INTEGER RANDOM NUMBER 
C SUBPROGRAM NONE 
0002 OIMENSTION 2(N) 
0003 : 0O 20 M=1,N 
0004 IX=1X*65539 
0005 IFCIX) 59696 
0006 5 I[X=1X+2147483647+1 
0007 2 6 Y=IX 
0008 Y=Y*.4656613E-9 
0009 : 20 Z(M)=Y 
0010 RETURN 
Ool1l END ° 
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FORTRAN IV € COMPILER VEOUT 09-15-71 15:55.50 PAGE 0001 

0001 SUBROUTINE VEOUT(AsNe¢NUMH, TITLE) 
C PURPOSE PRINTS UP A VECTOR 
C A INPUT VECTOR 
Cc N LENGTH OF A 
C NUMH NO OF CHARACTERS IN TITLE(MULTIPLE OF 4) 
Cc TITLE TITLE OF THE VECTOR 

0002 DIMENSION ACN) »TITLE(20) 

0003 100 FORMAT (/41X,20A4) 

0004 101 FORMAT(1X,10(5X_12,6X)) 

0005 102 FORMAT(1LX,10E13.5) 

0006 NN=(NUMH43)/4 

0007 WRITE(6,100) (TITLECI) »f=LyNN) 

0008 M=N 

0009 IF(N.GT.10) M=10 

0010 WRITE(6,101) (Ly1=l)™) 

OO1l WRITE(69102) CACTI) »I=L9M) 

0012 IF(N.LE.10) GO TO 30 

0013 WRITE(6,/101) (IyL=LiyN) 

0014 WRITE(6,102) (ACI) »1=114N) 

0015 30 RETURN 

0016 END 


TOTAL MEMCRY REQUIREMENTS 000354 BYTES 
152:55.52 17.673 RC=0 


1000 39A9.— (| OR eREaI 


Ds en . 
(80 Se i 


222%. 


APPENDIX A.2 


EXAMPLE OUTPUTS 


RELO] : Votaw-J&reskog Example Data 


RELO2 : Load-Novick Item Parameters 


| S.A 0Qwa49A | 
; - eTuaTuo s49HAKa ee 
- z ; tre th fi er aN ve ; 


6360 ol tqmsxs gorzortloweso¥ : 
ork | 
21930m6 164 hast: Aivell-beed yf vd 


7) 
7. 
- 
ry ; v2 7 
: , 7 ae 


223. 


IWHYON 3YV SNOLANGIYLSIO 3vOIS Borys 
IWWYON 3YV SNOTINGIYLISIG 3xOIS INL 


0s0°0O W3A317 JINVIISINSIS 
84 SIVAYBLNI SSV19 JO ON 3H1 ¥OS NOI1d0 
Z VINNYOS NOILVWILS3S YOS NOILdO 
T 101d YO NOI1d0 
6) INdinO avyvd vos NOIdO 
6666 UIAWNN WOONVY YI9IINI ONT LYVIS 
i 3YODS 3NUL NI SYyOLIVS 4D ON 
2 SiS3i-Luvd 4O ON 
OT 31dWVS HOVA NI SL9AFENS JO ON 
0002 Q3ZLVINWIS S31dWVS JO ON 


AWHWYON *4=fOT=1000Z=N491TY3N39ONOD ‘V1VO JNdNVX3 IOXSIVOF-MVLOA 


224. 


Zl61S°0 


@L%T6°9 


=4330) 
20 
40) 
(40) 
20 


244330) 


ALT3N390WOH 
3LOL8Ic°O 
3S%002T °0 
3907T2T°O 
3TZOL OZ °0 
pe) 


ALI 3N390W0H 
3602S02°0 
3S4002T °O 
390*%1TZT°O 
3120202 °0 
8) 


ZO 3TLETZ°O 20 36€L7Z°O 20 32078Z2°O 20 3TL0S7°0 


rAd) 
zo 
10 
rAd) 


TO 


4 € 
0°0 0°0 
» € 
20 341LZz2°0 
€0 3S2z802°O 
eezte°o 
SZTEB°O 
6L6£9°O =4430) NOILVUNLVS 
3S4%002T°0 20 34041ZT°0 
306€L2Z°0 10 300Z0TL °0 
3002012 °O ZO 3610Z82°0 
3SOTIZT°O 20 39 %ZZ1°0 
€ -9 aes 


s09¢€6°O =43309 

367002T°O rae) 

30S2202°0 fe) 

J00Z0TL°O age) 

3as0T1I2t°Oo rAe) 
Coe aD 


3B8TISTT°O 10 34%96€°O 


¥ 


0°0 
S) 


NOI LVUNLVS 
340¥T2T°O 
300Z20TL °O0 
J0”¢8TL°o 
39L%c2T°O 
Ga, 


Z T 
Si¥VvVd JO SIINVIUVA 
0°0 0°0 

rd T 


$193443 O3aXI4S 

0°0 8 =NVAWS 

JINVIYVA 3YOIS YOUYs 
BINVIYVA 3YxOIS 3Nul 
VHd W 

ALIVIGVI134 
SYZLIWVYVd NOILVINdOd 
sNOIS ¥3dSIG 3JYOIS Luvd 
ZO 3129202 °0 Xe] 
ZO aSOTT2T°O OR 
CO ASL Zila ra iI 
20 39919S2°0 t = 


1 Ss 
XTULVR NOIS ¥3ds 10 


sNOIS¥2dS1Q 3yOIS jANYL 

ZO 3120202 °90 Fea 

20 3SOTIZzTt°o (Sa 

20 39L%22T°O By) 

20 3648802°0 rr. | 
9) 


XIYLVW NOISH3dSIG 3¥ODS 3Nel 


€ 


TO 3000€S%°0 
To 3000S92°Q 
TO 3000892°0 
TO 30092S%4°0 
Lo 

XIYLVW ONIGVOT wOLdVS 


ANOS 
' 
> 4 


TO 3L%8S%°O 10 366%02°0O 


2 T 


YOLIZA SNOILVIARG GUVGNVLIS wOUY3 


0°0 
£ 


(9°Ss4) 


0°0 0°0 
2 T 
YOLIZA SNV3h 
ViVG 3H1L YOS LVWUOd 


CELCRCLCELCERPELESP ELE CEL LCEPELCELCCELCES 
SYSLIWVUVd NOTAVINdOd LNdNI Y 


e 


CCCECECECEECECe 


PECHCEP2ZOSESCECCCECEEL 


\ 
ys, 
2 
Wy 


8s O*SeS0SE 0S 0° 


SSkaeE OS O*STSATE OS 


225 . 


TES90°O =Z1VWIIS3 


Z66€9°O =1VS/WOH 


ZO 36866ET°O 
TO 31SL6%S °0 
00 3L€9T2S°O 
20 3968TIT°O 
2 2) 


4661L°O =(VAONV)1S3 13Y G3SVIGNN €66€9°O =VHd IV 


TO 3YTLEEY°O— 
TO 328L2ST°O- 
TO 3942S8S°0- 
TO 3T00%0T°O 
TO a£6€292°0O 
10 3%8%SlS°0 
TO 3€TEe*2T°O 
10 3¥z88TE°O- 
TO 38€8S7E°0- 
10 3%68STE°O 
oe =) 


96L0T°O =31LVWI1IS3 
29L0€°O =34309 ALISN390WOH 


TO 
To 
Ade) 
tO 


00 


“00 


10 3122247°0 =3 TO 3068986°0 


00 
ate) 
TO 
fe) 
00 
00 
00 
a <0) 
TO 
TO 


TLECO°O 


31S26%S°0O 
3L0T892°O 
3€44%"T°O 
3681095 °0 
(F 8) 


3281S%7°0- 00 


4 


J3T69ET°O- 00 


9 


zL08%7°0 


¥E4TO°O =VAONY YAGNN SSLVWILS3 VW3YN G3SVIBNN JO 3JINVIUVA 
=VAONY YZGNN JLVWILSS VHdW JO 3JINVIUYVA 


=44309 NOILVUNIVS = NOISYAdSIG JIdWVS 
09 3L¢9T¢S°O 70 3968TTT°O ve | 
U3 397791 50 TO 3681095 °0 = =3 


COmaTO SELLE SO TO 3€S€S02°0 Cane 
To 3€S€S0¢°0 ZO 30S04%LT°O iG Os 
4 <3) oe 


XTULVW NOISY3dSIG 3TdWYS 


00 340641TE°O- =NV3W9 
Z3TLZ106°0— 10 3%STST°O- 10 3E609T°O 
€ 2 T 
YOLIZA SNVIW 3IdAVS 
30898S6°0— T0 3S0072T°0- 10 32%26T °0 
€ Z TU 
YOLIBA $1934433 G3XI4 JIIdWVS 


=3SW 20 3LSSE8T°O =8SWH 720 3080%L2°O0 =VSW 


32502 %6 °0 TO 3¥Z26SLE°0O- 10 3E9EETY°O-— OT —¥ 
307¢S6e72 °O— 10 36€ETBT°O 10 31160S2°0- 6 <—¥% 
300822T°0- 09 3976849°0 00 3808L0€°O Su 
3SECELT°O TO 394%60%E °O TO 39%78222°0 he 
3470 %E°O TO 31020%%°0 TO 3€8SS21°0 Ss) 
3S84%8S€°O 00 3202688 °0- 10 3TTSEvE°O Saud 
B0E92TY°0— 10 39ZTEEB °O- 00 396€ST8°0 5 <u 
3S€9£9%°0- 1d 3Je79I7 tt “0- 00 3c€vlL8T°O0- € —4 
3S92819°0- TO 361T1T¥S°0- 00 30%26S9°0- 2 <u 
3247f0%7Z2°0 10 392068S °0- TO 3669868 °O oes 
‘S  24) (a 3) <2) 


X1IYLVW VIVO 


eeeceerce ceeceeecezee 


e SNNY 3 1dWVX3 e 
eeerererereececrecce 


_ - SS a 
ee ee 
> 


et in 
7 ia oe UGS 


an > - 10 3eEBeSe.0- 10 32a tact fp 

my é se =o —~ : a= ieee BeEstas. Be 
- Sa = ee ts) 7 seis eee”, jo. 
Pa . ; ) 3 2} Lee er a Sorgen te '~oo 
pan ro ave 
7 — 7 ; : i = . = = 
Se yes . . he: Po oe $51.0- : 
~ ' vas Lit ECts9o- ie 3PkL LAS 
7 a Ss .. S ! BOSH of Seett 6 
e —— PRBETSO <tAVOMRITZa 439 cazalaty KOvEa.o “AHAUA So BUSNTTSLG a “so 3888s. 
eet ie : E hs ; . 0 
tS ae ee ai a em 
ne I < J é . ay ; 2 . or 5008880 bo 32008t;0= £0. 
a ere ! 5 's oa ree 2d 
ae pees: ~ a5 4 = os 7 
es? : oF ee ) 88a 00 wae ii 1953 
F aes | | xeg5 —s 2 3) ages i 
~ a | | ier <a “2 a5 Ca, =e) gy “ae 
“in . . + “So gegsei.o . 10 gone a wigeezeoreo 
— ) 5 OG 3t€atse 0 LQ SEdeEOI.O ay 
i | ; | 10 3f2kesk.0 «= AGT OgatD eon Sis. phen 
255 ; | SC S¢8VCEL.0- 10-312TRa@sO 00 ELAS © on 
iP . Se€?£a.0 =TAZNMOH SéTOE.0 49309. YTISK3D0MOH ~~ -ST08A,0 © =79 903 VOETAAUTA2. : fereaiia 2fKA2 ; 
ee ) é a ACTOL+O SATAMFIZS - PTESOIO =AVOWA) AIOMUSETAHITES AHDIE AO SIVATRAY 
7 . F J£280.0 “3TAMET29- ce Ss 0 sAVGMS AIG 230 ANETeS 438 Caza Tan A 334A HAV =A 
4 i , = 
i - 
— : 7 
- x 
_ 
. i 7 
a 


226. 


71300W 


00 3S6829T%°0 00 32E9827°0 | 0°O €9-3900949°0- + 
10 30ZTeTIT°O TO 39Z0ZIT°O { 0°O T0-36696SE°0- € 
10 36TIO¥T°O TO 3590%%t°0 | O°O T9-3€86S22°0 2 
00 3221996°0 00 35S022S5°0 ! 0°O T0-3€%S041°0 T 
03193dx3 JINVIUVA | G3193dx3 NV3SW Luvd 
°4°w YIONN S3NIVA G3193dX3 GNV SSLVWILS3 193443 OIXIFS YOS SIIASIAVIS JATIdIUISIA 
10 3652278°O0 20 3S%L9ET°O { 20 3629S50T°O 20 3T809TT°O yours 
20 30€S%4L°O €O 38E8TZI°O 1 20 36L9S0T°O 20 3S2Z8IT°O AS3Lluvd 
€0 340STL8°O €0 3¥TELIB°O 1 20 30%2979°0 20 34%799029°0 Loarans 
03193dX3 a3A¥aSg0 03193dX3 G3AuN3SSEO 
JINVIUVA NV3W 


IZQOW VAONY Y3GNN S3NIVA G3193dxX3 ONV S3YVNDS NVAW YOs SILISTIVIS JATIdIYISIG 


TO 3€E8S99°O =IWAYZINI SSVT1D 0°O =WNWNNIW €0 30000TE°O. =WNWIXVW 
CPECEPE TCP VP Pe2VeCeCeECCE 

; © iINdiNO 30 AYVKWNS C 

CECVPLELL PLELLELELCECLLE 


e 


40 3€&: 248 00 sansa 
$300 Sabi 0 
sae oe, 30M 25038 0 oat: 


¥ ; nO 7 a ; : ) dy pes seu souataay eS 7 
_ ; ~ a : =i : ’ . CITIZAKD ae _, S3ve280 | 
: _ | can J a £0 pre 24 - Soesigie \ 
aaa h $0. 3868. 
a . | 10 BRESTS840 80 3#eTaEL. 
=o om . _. J300M .3.m ABOMY 200A aT 338K3 | “oath 
= . es . : G3T D3 : 
— aug ; = Pe | ce Of sasidetse 90° 
7 : | | ss 2PSEOAELG £0 ale . 
a : : 2 ee Ae [GAOT LNT 20. 40° F8SOS EEO 
») a een, 0 ee 7 . | 66 Beetalpso OO 38E8GS>-0 
mi a O « cae : ; . 
ao as SL r ; = 
oak! oe a “ 
Ps Fs < 7 i rs = ~ 
es : + 
ae 7 . ‘ ' 
a =" ; 
=. * 
x : : 
ie: 7 = 1 - 
: a 7 / j 


se 


i 


Ss > - 
id 

8 7 _ _ 

os : : is 

' = 7 a - 

: = _ 
Pas 

oy - b. . 

_ re : _ 

i 4 


Pages (A 


€0 JOOOTET*O €0 30S618T°0 }3=— SS” —¥ 
€0 3000S0Z2°0 = €0:312SSEZ°0 a -¥ 
€0 30000TZ°0 €0 30L91427°0 = ES -¥ 
€0 3000961 °0 €O 3€0102Z7°0 9-2-4 
€0 3000L$1°0 €O 349088T°O Te -¥ 
€0 300029T°0 €0 32ZSST°O0 §9= 0% 8 
€9 JOOOL4T°O  €0 3L08SZ1°O 6€ -¥ 
€0 3000901°0 €0 36L0TOT°O = BE -¥ 
€0 3000S0T°O 20 3569608°0 LE -¥ 
Z0 30000%6 °0 Z0 32T6859°0 §86«9E 8 
20 3000009°0 «= 20:«»3€Z2¥1zZS°0 = SE -¥ 
Z0 300006%°0 Z0 311902%°0 = EY 
Z) 30000€E°0 20 F€L89%E°0 EE -¥ 
20 3000062°0 20 3299LL2°0 =. 2€ 8 
20 30000€2°0 20 38S€L22°0 ~=osdTE:s 
ZO 300006Z7°0 = 20: 399TZ8T°0)— «OE = 
ZO 300008T*O 20 39684ST°O = 6-4 
Z0 300009T°0 ZO 31S88ZT°O }8=— BZ 
Z0 300000T°0 ZO 3TELLOT°O «=k = 
TO 3000009°0 10 3680S06°0 92 -¥ 
20 30900%1°0 10 3000%9L°0 = SZ _—=¥ 
TO 3000009°0 10 3208249°0 42 -¥ 
10 3000009°0 TO 36891SS*°O 8=6€2 —=-¥ 
TO 3000009°0 10 31LLTL4°O )=— 22 = 
19 309000S°0 10 3190S0%°0 12 -¥ 
10 300000€°0 TO 391T1T64€°0 02 -¥ 
10 3000008°0 TO 32S6T0€°O ot -¥ 
1) 300000€°0 10 36712927°0 = 8T = 
10 3000001 °0 10 38628Z2°0 = LT -¥ 
10 300000%°0 10 3€L4661°O = 9T = 
10 300000€ °0 10 30Z284L1°O0 3=— ST 8 
TO 300000$°0 TO 3€L9EST°O §=— oT 9 
0°0 10 369%SET°O ET -¥ 
19 3000002°0 TO 34€L6TT°O 21 -¥ 
TO 300000T°O 10 302T901°O ~=sdTT:S«-¥ 
TO 3000001 °O 90 3L9L24%6°0 OL -¥ 
0°0 00 316S6€8°0 6 -¥ 
TO 300000Z°0 =: 00:-«35694641°0 =U 
10 300000T°0 =©9 00: 3255029°0 ak =u 
10 3000001T°0 00 3€S0109°0 9 -¥ 
TO 3000001°0 =9: G0 :-ALETO¥S*°O «OSs = 
10 300000T°O0 00 435S5299%°0 4 =e 
0°0 00 32S%8€4°0 € -¥ 
0°0 00 3252796€°0 2 -¥ 
0°0 00 310z8SE°O Tt -¥ 
z -9 t -9 
ALITIGVI13¥ 3O NOSTYydWwOd 
TO-3ETTLEZ°O T0-3ZE06TE*O | 00 3¥€0€82°0 00 3€8S09L°0 40D 13u 
03193¢4x3 Q3AN3S80 | 03193dx3 G3Au3S80 
JINVIYVA NV3W 


S3LVWILSS ALIVISVIISY YAOA SIIISTLVIS 3A11d1¥9$ 30 


TO0-30000S2°0 9 =IVAYZINI SSV¥19 00 3000032°0- =WNWNNIW 00 3666666°0 =WNNWIXVW 
TO 386€S2%°0 =7°O- M0139 SILVWILS3 ALITIGVIT3Y JO AININDSYS G3193dx3 
(G3SVIG*VAONV )VINWYOS VHd lv NO G3Sv9 SI NOLIVWILS3 

CCE CECUCE USC eeUCTroeeruccee 
e ACNLS ALITIGVII3Y e 
CECCZSCEVeUl ere teeeeccveece 


aeis ; : 
—_ = 7 i 
Patel —) 
7 y =a. * S— 
_ : = > . “aa - _ = - ~ 
ae a == 4‘ 
a... = : f 
: 4 : 
—is o 
a f ‘: 
Se 
ae) 
— - i { 
oy, = ; 
= | 
“a * 
| 
Bas 
is 
wt 7 


wt 


al 


~~ 


Se waret 


ei al 


— 


j 


s “R000091.0° 
ae aceens =’ 
0001.0 


|= “3900005.0 60 3822@0Te.0 4 


3 267860266 
IY 3000006 .0 


00 p= ty 
a ) ae108 be 
00 3£20100.0 ra 


oy sans << O0-303s¢aT 20 Fin 
wa 0-0 00 3iPePee.o | nn 
io ae OO BTDTSee. 0 + a 
- SO00G0i.0 iG 30Sfa0/.0 # 
7 eee e. el ae 
z & — 20 
io 300000¢.0 a rasa ¢ 
10 2000006.0 10 -30SBeTF.0 2 
1G FOR 0006.0 10 JET Sees 0 ‘ “F 
10 FOCHOOER.0 40 B2BPSESS~6 1 i 
13. 3000908.0 Td 3e5158a8.0 = 
7 \go00098.9 16 30ePf0E.0 Of <2 a 
30000040 16 3011e8E.0 05 ok — 


rO 3440a08.0 I[5.-A ? 
Teer bed SS -H 


io 300000B.0— 16 arsoiee.0 ES <A ) - 


iO 2006000.6 
$O 32650081.0 
fo so08000.0 
So s00000f.0 
So JOGOGSs *Q 
£0 50G0 Chi.0 
So 3po0nes.0 


SO 3000GES.0 


oo s00Gpre..0 


$2. SGOO0EE «Dd 


SO J0000e+.6 
$0 3900006.6 
SO 3uSCOVe 0 
€9 30C0F01..0 
€0 300080! .9 
£0 2000TH4a0 


&6 pete | _ 


£6 309 


EO an0deeL oo : @ 
80: Lo 


0208 


a donouei-o 


10° 3% Settee Oo oS =A 
IG 306034730 S of 
fo. 3280¢5e.0 oS A 
$6 31é7091.0 ¥S <A 
SO 3£268S1.0 85 ~a 
SO J886h4 E.G os =f 
$O ade(Tal.0 of -<8 - 
SO S82ESS.0 SE =A 
SO aV@StYS.0 SE A : 
So 36T@002.0 €E -A2 : 
$8 21fe054.9 ef fi a 
SO 3ESei5250 - @f a o 
; BSiG0G60 BER 
o 


228. 


98T226°0 =(1S3)9 waddN 


$00S26°0 


=QNN08 w3addN 


¥8189%°O =(153)8 Y3M0T 


O€T+IS°O 


=ONNO@ Y3MO1 


20%°% 


88 


=@ v3addNn 
OIE 


NVHL ¥Y31LV3Y9 


=340 


°6 


$ 2S2°9 
=Vid 


S2@t =@ Y3MOT NVHL SS37 S3SVD 
0S0°0=(HIV3) 13A3 ISIS S3LVWILS3 


0°0 
ZO 300000T °0 
20 3090082 °O 


00 36S2001°0 


ZO 3S98%ETPO 
20 34%6£598°0 


30 ON 
VWHd1V 
By <8 
Lou 
rae 


= 0°83 


= = 
7 Fe as: a : MD= 0° 


fevew Leen) 


= 
a} 


- 
on ar eF 
co eel oi 


oe rut , 


one 


S ; ; £ } 
y we 
 - & 
e 
= Si 
: 7 
i a) 
7 = = 
< iy 
. j a: J 


8 


ah 


229+ 


O°T 


* 


(ttle tate tt tal 1 tl = la lal lath at l-lif te 


on 


6°0 


8°0 


L°0 


9°0 


0000°0 


INIOd ONIddVTU3SA0 (+) 26343 GIAYISEO (s) S03us 03193dx3 (°) :31L0N 
2202 £°0 2°0 T°0 0°0- T°0- a 0— 


$°O 


=80ud 


+ + 


Of 


* 


734 
++ +4 * * * | 


| 

I 

I 

1[-000°2 

I 

| 

| 

| 

{-000°% 

i 

i 

{ 

! 

1-000 °9 

| 

| 

| 

| 

1-000°8 

| 

! 

| 

| 

1-000 °OT 

I 

| 

I 

| 

1-000°2T 

| 

I 

I 

| 

{-000°4T 
0°91 

-000°8T 


\ 
{ 
| 
| 
I- 
| 
| 
{ 
| 
| 
{ 
I 
I 


| 
1-000 °02 
(ZIAININO ABS 
=JQN cO 340%T88°O =IHD 21831 L143 30 SS43NQ009 OS-IHD 
SASVEIZSSSEEELSSSSSEEVAELESLFLLSSS 


% NOTANGIYISIG AININOIYS JO 101d & 
SKELSSELLSISZSEALTSSELELALERESLSLELE 


jg 
e 
a 2 oe 


-- 


ta 


J 
> 4 


a 
=) 


' 


/ * . ~~ x : af 
* : " 
Pa 7 =a 
: : : 000% 
wi a > 7 
SS tate 
4% | 
. td i“ 7 
. . 7 1 / 
4 F : i. On* le 
os 1 .* | en 
. | 
© EF. 
. i 41-0004 
4 #£ * ae ee 
» * me oe 
Z “ : Pa eee * 
& + + »* + 4 * * » 
BP fae ss se sa es dt se -15f-1- “bile -i- wandat -\-f--tsi 
; Osk C40 6.0 ° ¥.0 rs > €.0 *.9 £.4 Sa a Ie a 
THIOG OWISGA IABVO {4+} 20384 rey (9) 20388 0939 
Ps <1 = ra 


4 Be 


230. 


€) 300065Z°0  €0 3680T0E°0 S¥% -¥ 
€O 3000SS2°0 €0 32€5¥0E°0 944 -¥ 
€) 3000892°0 £0 39256S7°0 E> -¥ 
€0 3000TTZ°O  ©€0 3425502°0 2% -¥ 
€D JOOOSLI°O £0 38669ST°O Ty -¥ 
€0 JO00LET°O £0 3Z%E8TI°O 0% -¥ 
€0 JO000ET*O 20 3259688°0 6€ -¥ 
ZO 300008L°0 20 3L2€TL9°0 §=8E -¥ 
20 30000€S°0 20 368950S°0 LE -¥ 
Z0 3000065°0 20 32%006€°0 9€ -¥ 
20 3000082°0 20 37S010E°0 SE -¥ 
ZO 300008€°0 20 3€Ih¥Ez°0 ve -¥ 
Z0 3000061°0 «20 31Z2T#8l°O = EE = 
ZO 3O0G0ET*°O 20 36%8S%T°0 Ze -¥ 
Z0 300002T°0 «= Z0:«-36L¥9TT°O =—dtTE = 
Z0 30000ET°O TO a¥TSLE6°0 0€ -¥ 
TO 3000006 °0 10 3z01092°0 = 62 -¥ 
10 3000002 °0 10 3409029°0 =z -¥ 
10 300000L°0 10 .326001S°0 22 -¥ 
10 300000%°0 10 3¥681Z¥°O 92 -¥ 
10 3000009°0 10 3%20TSe°O Sz -¥ 
TO 300000%°0 10 3969€6Z7°0 9-4 
10 300000S*0 10 36%02%2°0 €2 -¥ 
TO 3000005 °0 TO 3TEee0z2°0 «=z -¥ 
10 300000T °0 10 3S5€L21°O 12 -¥ 
TO 300009Z°O 10 309€TST°O 02 -¥ 
10 3000001 °0 10 32TL521°O }= 6-8 
10 300000T°O0 =: T0:-AHO9TIT°O § 8t -¥ 
10 3000002°0 00 3926£96°0 §=LT -¥ 
10 3000002°0 00 392LS€8°0 91 -¥ 
0°0 00 3290L22°0 St -¥ 
10 300000Z°0 = 00-3682 4E9"0 = oT 8 
0°0 00 34S1S55S°O” ET -¥ 
0°0 00 318z788%°0 2T -¥ 
10 300000T*O 00 3L9TCE¥°O IT -¥ 
TO 3000001°0 00 36€008E°0 OT -¥ 
10 300000T°O 00 32499€£°0 6 -¥ 
0°0 00 34L6862°0 8 -¥ 
0°0 «00 34099270 ke 
10 JOOOO0E°O = 00 :-avaSLEZ*O O98 
0°0 00 3te*zTz°0 83S -¥ 
0°0 00 342€061°0 4% -¥ 
0°0 00 3S90TLT°O € -¥ 
0°0 00 3810%ST°O 2 -¥ 
TO 300000T°0 10 366L252°0- 1 -¥ 
2-9 1-9 
ALIVIGVI1ay JO NOSIYVdWOd 
10-36€4%E4T°O TO-3I8EEEETO | 00 365zTE8°0 00 3Z9LETB°O 409 134 
031934dx3 O3AN3S80 | 03193dx3 Q3AuN3S80 
JINVIBVA NWSW 


S3SLVWILSS ALITIGVIISY wOs SIIISITivis J3ALId1¥9S30 


T0-30000S2°0 =TVAYRINI SSV19 00 3000002°0- 


=WNWNN IW 


00 3666666°0 


=WNWIXVW 


IO 386€S24°O =Z°O- MOTSE SSLVWILSS ALIDIGVIIIY JO AININDAYS C31L93dxX3 

(CISVIGNN*VAONVINOILISYYOD JOLSIYN NO O3SVE ST NOLLVWI1S3 
eeeeeecere=zeeerecrcereree 
AONLS ALITVIGVIIW3Y eC 


e 


COCCEC ECS 


BOGED CAAA 


oDveve 


Cece 


Pe soot iawn 


4) 


3 erates 
—_ _* Saar 


we 


Sf ae 


1 “aREoEe fy $0-38EEGC1 «6 


~ a 


03723943 
00 3easi£e.0 


3 


T3399x3 


er 
$s =3, 


fo 2000051.6 


0.0 
620 
0.6 
0,0 
FOGCOSDE 0 
‘O.D 
O20 
JOC 000i .0 
3000G0! 20 
2600001. 
6,0 
o.o 
20009005 ) 


NGO 20f 30 
§u0ou01 f 
3O00G25.0 
qonoe iOL +B 
366.0002. 
300000e.6 
39090008 .0 
+0 


COGOC? «2 . 
3000907 « i 


FOVVOAS T.0 
70B00ce 3) 
AGOOGE 1.2 
FOOOOS 1.0 
GOOGRE 1.0 
agetees 0 
FOOCOBE +0 


s000e2825.0 


sopaeeeso 
20D 00E 2.0 
3000080 «0 
300008120 
3O90TEL <0 


) 300085 1.0 


FOOTIE « — 
RODGRSE 


906 0825.0 —— 


“BO0CP2S.0 © 


00 35 = £18.00 


srvaiecia : cae 2A 


navas2ec 


ea 


10 
pets) 
60- 
06 3TTEOCL.0 ¢ 
oo Ji€ss75.0 @ 
OO 3682TES.0 2 
T 
8 


oO 328 0a8s.0 
GO aTTeBeS.9 : 
OO J¥#dehe.o 
OO 3°f008F.G 86 
OG S13'Gke.d 8 FL 
oo 3495685 .0 SL 
oO aeevede.G. ef 
OO BP6TSE3.0 AT = 
OO ST@0TS¥.G 21 : 

OO 4adtV28G.0 ef a |: 

QO Zaseear.a FL <8 

10 34GaiiT.c BL -®. 

$0. FSpvesi. 8060l MA 

fO 364¢121.0 O05 =F 

@ FJaveTe sO i =A 

£0 3LESHSS.0 §& ~R 

IG 2800Te#5.0 ES =f 

IQ 3cPAOEES.0 45 =A | 

1D 3eS02CE.C as =8 Es 

{0 BPSIS4.D o§--A 
10 3FeGome@.o TS 8 
10 300303840 8S -f pa 
EO 389f0a5.0. 5S -2 

10 3AFet £20 
£6 3fVeeti ad. 
&0 Jest2ar.0 
SO 2i5i08reD =F 
§0 SCiPFES 0 4 es 
80 382010849 
SO 34°00RE 


- 


231. 


8256€6°O =(1S3)98 Y3ddN 


Z€898S°O =(1S53)8 YIMOT 209°% 88 
1LZ91%6°0O =ONNO08 w3ddN 


2Q Y3ddN NVHL Y3LV3RYD * 2S2°9 GZ1l =@ ¥3M07 NVHL SS37 S$3SV9 30 ON 
10OTZZ9°O =QNNO0G w3MO1 <LLLA =340 °6 =v i0 0S0°O0=(HIVI)TZA31 IIS S3LVWILS3 VHdIV G3LSNFAV 
TO 300000T°0 00 3%%0Sel°O Dye el 


20 300000S °0 ZO 3LL6L647°0 Lv -¥ 
£0 3000S¥T°0 €0 3406002°0 a | 


a ye 7 — - “aust 

; a — ae 
see fThbty Pte? ese 
i pth Mo ; g " : 
=v Cir ia eae 


tec +75 : 
eS i MON we 
7 ide By : 


3 


Lis 


_— 


mew etee 


; 


ee 


282} 


L8OZTLZILL =XI Y3EKNN WOONVY C338 IS Vlxeoeeee 
LNIOd ONIddVIUZAO (4+) $0394 QFANISEO (#) £O3Ys GILIIdxX3 (°) :31L0N 
o°t 6°0 8°0O L£°0 9°0 s°0 ¥°0O €°0 z°0 US) 0°0- T°0- 2°0- 
b<bode del -d-f-l tb tlh tbe te t-te ett ttt let f-f-f- feta t-te ttt f-t-f-t-t-t-t-t-1-t-t-f-1 13u 
oe e+ + + + ee HR I 
++ * ® 


af 


1-000°02 
(% YAININOIY4 
0000°0 =80ud S2 =J0N ZO 32282S8°O =IH) 21S$31 114 30 $S3NO0009 OS-IHI 
SFSSSSTLLTLETISSSSSSSASSATATIAATIZ 
% NOIANGIYLSIO AIN3NO3Y¥I 3D 101d % 
LLLSRLVELSLSLSESSLASSSSTLALLLATIZEE 


* any ue | 


eee 


| © ii. eee ' - - ie 
+ pee [nI- tele I-lobetafetotstad-tate 
+f=fsisi=i=t-1Sk ~bedstade tote le betta bt fel att tittle 1 > “ 

. . ; e.0 8.0 T.0 3.0 Ay — = ‘he “Ga= bk 


verte QAWGAIRTVO (4) Daa 2ag ad AR 
. eer es ry 


rt 


TO-309ZTT°O TO0-388192°O TO-3T9TOE°O 10-3147259°0 


TO0-36S8S6S°) TO-3TSBI%¥°O TO-3ESSB8%°O 2O0-3SEE%8B°0 
6 8 L 9 


TO0-3S€€2S °0 
S y € @ T 


an SJINVIYTADD W3ALI W31Wuvd 
mY 78649 °0=029y 92€99°0=134 TO 3888%ET°O =uVA BOYS TO 36569992°0 =¥YVA 3NYL TO 32%S00%°O =xVA 10 3001L%%°O =NV3W 
V3Z00W JAISO WaYIN Y3ONN SYZLIAVAVd NIILVINdOd 
TO 3S62%6€2°0- 00 306969S°0 TO 3STS8IT°O- 00 3920%0T°O 00 3090288°0 00 3900S6%°0 6 
10 31 7%6ST°O- 00 3290S629°0 0O 39£0S%8°0- 00 366€6ST°O 00 3000T08°0 00 3J0000€S°0 8 
00 3€¢28S6°0- 00 306214¥S°0 00 3¥1TT9S%°0- 00 3%206T2°O0 00 30009L9°0 00 390909L%°0 ra 
00 38%606¢°0- 00 3L26¢€8°0O 00 3L07298f°0- 00 3%92S442°0 00 3000%LS°0 00 32000%9°0 9 
00 J3€L6TZT°O 00 320€0%2°O IO=ARELS CULO 00 36S16%2°0 00 3000TLY°D 00 3900S6S°0 S 
00 3002622°0 00 31999€2°0 00 3298S9T°O 00 3%479S%2°0O 00 3000¥%E%°0 00 3000£65°0 % 
00 3254092 °0 00 38£8959°0 OO 388%lT¥°O 00 39SL€22°0 00 JO008EE°O 00 3)906%S°0 € 
TO 32S8Z1T°O TO 366820T°O 00 3L€0S%8°0 00 366£6ST°O 00 3000661°0 00 39002TL°O é 
TO 3T0E992°0 00 359129S6°0 TO 388%0€1°O T0-36€8298°0 T0-3000096°0 00 330006 °0 T 
XJONI °45IG YIMOd ° ISIC °SNOD SAYHL JINVIUVA AL1N3143510 yO SIG WIL!l 
SYILAWVUVd WILT 
00 362)S%2°0 6 <u 
00 30S€292°0 8 <-u 
00 3029S€2°) | me 
00 30039T€°0 Sat 
00 352S%62°0 SH 
09 3SESE62°0 pI) 
00 3SSlTL2°0 (Se 
00 3S1T6%SE°D cou 
00 30S6S2%2°9 1S 
eye 3) 
00 30S€292°0 00 30¢29SE2°O 00 30089TE°O 00 362S%62°0 00 3SES€62°0 00 3SSLIL2°0 00 3ST54%SE°0 00 30SS¢%2°9 6 <3 
00 3006082°0 00 30822S62°0 00 30026EE°0 00 30SESTE°O CO 3062%TE°O 00 3946962°0 00 39TO008E°O 00 30026S2°0 8 <-¥ 
00 30822S2°0 00 39149922°0 00 3049%0€°0 00 3022€82°0 0O 389¢282°0 00 3¥2ET92°O 00 32621%E°0 00 30%2EEC2°9O bat 
00 30026€£°0 00 30%9%70€°0 00 300960%°0 00 300808€°0 00 302S6LE°O 00 JO9ETSE°O 00 30888S%4°0 QO JV09ETE°D Cys: 
00 30SESTE"O 00 30¢2e82c°0O 00 300808E°0O 00 3S204SE°0 00 3SEBCSE°O 090 356S992E°0 00 3S199¢2%°0 00 30SST62°9 S (=u 
00 3062%T€°0 00 3892c¢82°0 00 302S6l¢€°0 00 3S€B2SE°O 00 3649TSE°O 00 3LSSSze°O OO 3181S25°0 00 3016062°0 > <u 
00 3016062°0 00 3%2€192°0 00 309ETSE°O 00 35S992€°0O 00 345S5S2€°0 00 310¥T0e°O OO JEL9C6E°)D 03 3010692°0 St 
00 3J0TOO8E°O 00 3262T¥E°O 00 30888S%°0O 00 361992%°0 OO 3181S24°0 00 3££9€6C°O 00 36804%1S°0 00 J0EETSE°O Cal 
00 300L6S2°0 00 30%2EEC°O 00 3009€TC°O 00 3056162°0 00 302S062°0 00 301T069¢°0 00 30etISe°O 00 30015%2°0 T oad 
See) j=) CP 8) SS &8) we =O) © <=) (58) To as3 
XIYLVW NOTIVISYdO) JIYOHIVYLAL WILT Y3INI 
(S$°S36) SY3L3BAVUVd WILT YO LVWYCS 
WwWYON J3YV SNOILNGIYLSIO 3¥053S sOUuY3S 
WaAYON 3YV SNOTINGIYLSIO 3¥09S 3NdL 
0S0°0 T3A37 JINVOISINSIS 
84 SIWAUSINI SSVI3 wO4d NOITIdO 
@ ViInNwWyG3 NOTAVWI1S3 ¥YO3 NITid3d 
T 101d YOs NITId9 
T iNndinNd Juv yod NIT1Ld0 
dik YSBWAN WOINVY J3a3IS ONILYVIS 
6 SW3LT 30 ON 
ST 3JIdWVS HIVS NI S193FENS 40 ON 
oooT GILVINWIS S3IdNVS 3O ON 
(ON )YOUYS*CONILNZIVI *6=F°ST=I1 SOOOT=N “SY3L3WVYVd ADIIAON GNV Gyo? 


= s - oo an a 
——_ See 8 é ' = a 
 «® SRBBGTees.5 - OO BONSEES.0 00 S00aL1E-0 
i BP -BOG0KE.0.~ OO BSeSisE.0 00 30RaRT+.0 
fs uc nae z07eDes. 0 00 FeSEISS.5 00 J0BETeE.O 
; GO HeGaLE.0 80 Wessas.c 00 FoOS#ers.0 
Pr a pbs athe | 66 2OSSEBS.0 Of 300808E-0 
=. AR OTSREE.O DO 304890E.0 QO 308°? .0 
a (00 WBESES.0 00 FOVESES.0 80 B0Hd00E.0 
+ he -GOTSMOEDAE. 0” OO FDBSSES.0 OO —FOCSPEE.0 
im BO" “BORESSS.0 00 FOSERES-G = 00 -B9OBBIE~O 
: a3 \ pes 5 x = . 
eae ; | 
Fe 7 
/ 
a . ' 
—_ 4 ’ 
x3041 27410 
10 3f0E39S.0 
a 10 Jvesy 17. 
Z 00. F5490eT .C- 
‘ 60 200FRTSe0 
, 66 BEveIS IO 
00 3Seeroes.o= 
GO 3e65B22.0- 
if) ALesedt.o- 
IG. 2¢Ge2ES20— 
2 @BCOS.0=0SR4 PSESS.0F IIH . 10 FEE 
w 
. 
= 9 a 
1O-30SS 17.0 fO-39e1eS .9 }90= 


i G6 


AIWF 


00 
19 
0 0 
66 
oo 
oo 
00 
oo 
oo 


‘ 
ZH10E.0 FX 


ge Taaga.O 
3¢208SE.0 
Feeetee 6 
3eg08F E.0 


G 300608E.0 


JOSSESL 
3O2ESPE.G 
FeS CPPE.WG8 


«J2%0 
3291 Sae.0 
3e28S01-0 
BHEGSPS.0 
Bieeoett.0 
ISDE COT LO 
aT SeSes.0 
50é% faced 
383902508.0 
q0eaed2.9 


INPEICS.0— 


‘98 
ob 
oe 
oo 
oo 
oo 
Os 
oo 
00 


i 2.) 
agor2des.0| 
a1Bi25e.0 
ay @eete.d 
FOOgLZELO 
a2fesee.d 
IOSSOTELO 
JAsssss.o 
FORLSTE.6 
REFSESS LO 


#2403 2 3Ant 


16 


JBHPCET.0 
af £0848 .0- 
FG SAT Le .0 
35 abP61,0 


{o=STETEST LO” 


he 3TO8081.6- 
Ref 1d2a.0~ 
306248 .0- 
(sereari. o- 


-O #RAV AGARS 


. aT ¥ 


4, 


£2640 


ih 


to Z@@e24S.0 


<¢ 


{0-2 2£ET?S 50 


De oo” 
a610008.0 
3E EBLE 
3800 108.0 
ae acess. 0 
Ze2aaSe.0 
IGae LTE. O 
goSE1AS.0 
S97Te0r$.0 
‘eet £75.0 


ROMA TAAY 
jo-3eeaTe8-0 
OG apeeeeE.0 
“00. 3d8TESS60 
80. FePSeAS. ie) 


s Fe 186.0 

7 3010e8s.0 
00 FOFEORS.0- 
B® Wee 1es.0 
00 SBRaE LEAD 


Behe 
— AdGRReH.0 


CO 3SPLI9E 0 SEES.9 | 
00 3610GRe.> oe OGT Pes. 
09° RALereE~0 an 2082S #540 


? = = 
O09 B27Ses.9 38 : 
7 Of JAseece.0 
OO 3eeTITS.9 F 


00 37asees. =O: 
O0.38Seas.0 00°30 <5 
00 e808 .0. = 6 ae ; 
00 38CEC2I.0. 7 J 
00 B8TO#OE.0 ae o red 
ee Sore ie heat 
_ —3300M ay100) JANAIA sap 2earseenae Da peeiaeys * 
=RAV 3URT £0 BS800h.0 = FAY £9, 200 1Te a ll 
| : palin 
Z u "23 9AAT BANDS 4att maine! os 
+ = s S i 7 ip. es : 
to-aeee22@.9 19-3F2810.9 49- see6si8 S032 E98-0 ; = 


234. 


00 30€€221°0 
10 3000001°0 
00 30S92491°0 
00 3S9€%61°0 
00 305242 1°0 
00 3I9TILZTI°0O 
00 3€1f64T°O 
00 30826ST°0 
10-3265 468°0 
8 =-3 


T0-320099T°0 
00 366€6ST°0 
T0-36€6992°0 
10-35 2LE8E€°0 
10-39S784E€°0 
T0-38898€E°0 
10-32 86182°0 
10-31 68€S2°0 
10-32 12s01°0 
3-3 


00 349LLTI°O 
00 306924 1°0 
TO 300000T°0 
00 3S6€061°0 
00 J€S¥SLT°0 
00 3€202LT°0 
00 39TEZST°O 
00 30%2TLT°O 
10-3€221S56°0 

i), 


T0-3008ZLT°0 
10-36€5992°0 
00 3%20612°0 
10-32 190%9°0 
TO-3EESL04°0 
T0-32 1066€°0 
TO-3€6TLZEE°O 
T0-J6S661E°0 
10-31 261ET°O 
ae) 


00 36S€8S1°0 
00 3S9€%61°0 
00 3S 6€06T°0 
TO 3000001°0 
00 3$56S42°0 
00 3780€%2°0 
00 3¢28sIz°0 
00 32S1942°0 
00 3¥EL9ET°O 
a) 


10-39292S2°0 
T0-3S2L€8€°O 
10-32 19044°0 
00 34%2S442°0 
T0-3£60209°0 
T0-38S1S65°0 
10-38 28%0S°0 
T0-38 96584°0 
T0-3£ 09861 °O 
8) 


00 36T0T#T°0 
00 30S24%L1°0 
00 JES%*LT°O 
00 36S6S4%2°0 
TO 300000T*°0 
00 34%9S822°0 
00 3214S02°0 
00 3TOTEYZ°0 
OO JE08%ET°O 

Sa) 


T0-3980L272°0 
T0-39S284€°0 
TO-3€€S104°0 
T0-3£60209°0 
00 36S516%2°0 
10-3965S96°0 
TO-3TT0S8%°0 
10-312 4484°0 
10-3 922861 °0 
8) 


00 3029ZET°0 
OO 3T19TTLT°O 
00 3€20221°0 
00 3%80€%2°0 
00 349S822°0 
TO 300000T°0 
00 3¥11S02°0 
00 36S9S%2°0 
00 31229€T°0 

ye Sa) 


T0-35%0022°0 
TO-38898EE°O 
T0-32T066€°0 
1T0-386166S5°0 
1T0-3964S9S°0 
00 3%4%9S42°0 
10-3610184°0 
T0-3401984°0 
T0-3596861T°0 

uP 8) 


00 JEEZ6TT°O 
00 JETE64T°O 
00 39TEZST°O 
00 3228S12°0 
90 321¥S02°0 
00 3%L1S02°0 
10 3000001°0 
00 3%600€2°0 
OO 3JE1EB2T°O 

=) 


T0-32S618T°O 
T0-3186182°0 
TO-3E6TLEE°O 
T0-382840S°0 
TO-3TT0SE4°0 
T0-36TOT84°0 
00 39Sl€22°0 
T0-3S4S4E4°0 
T0-3S08821°0 

BS 


00 39S9EZT°0 
00 30826ST°0 
00 39%2ILT°0O 
00 32519%2°0 
00 3TOTEYZ°O 
00 3559S42°0 
00 37600€2°0 
TO 300000T*D 
00 32€869T°0 

ce =) 


XIMLVA NOILV13ud09 WIL] 


T0-36926S1°0 
TO-3T68ESZ°0 
TO-36S56TE°0 
T0-3896S84%°0 
TO-31L5%84°0 
T0-3401984°0 
T0-3549S¥7E4°0 
00 356€65T1°0O 
T0-3841661°O 

Bo 8) 


TO 300000T°0 
00 30€EL2T°9 
00 3¥9LLTI°O 
OO J6SEBST°O 
OO 35TIT¥T°3 
00 JICILET°O 
00 3EEZETT°O 
00 39S9EZT°9 
T0-3929669°0 
Saas) 


10-3920669°0 
T0-3265468°0 
T0-3€27256°9 
00 AYEESETSO 
00 JE08%ET°O 
00 SILZ9€ET°9D 
00 3ETEBZT°O 
00 32€8591°0 
TO 300000T°0 

Yeoss 


00 392L3401°0 
TO0-329949T°) 
TO-3008LL1°9 
T0-39292S2°0 
T0-3989222°9 
T0-3S49022°) 
TO-32S618T°O 
T0-36926S1°9 
cO-3LEE4I9°0 

oVe=) 


fO-3LEE9I9"°O 
TO-3412S0T°9 
TO=a1 LST e ho 
TO-3E0986T°0 
TO-3527661°9 
10-359586T°0O 
TO-3S08e82T°0 
T0-38%2661°0 
TO0-36€8298°0 

Pcs 


6 <-% 
8 <-¥4 
4 #-4 
9 <-% 
Se =F 
7 <u 
€ <-¥ 
cou-o 
Tt = 
6 -¥ 
8 <4 
[Ee oe) 
ee | 
S$ -4¥ 
9 -¥ 
€ -u 
2 -u 
t= 
Y31N1 
6 <-u 
8 <-¥ 
LL =u 
987-4 
Sei-¥ 
VET =u 
5 oe) 
¢ =u 
To o- 
6 <-¥ 
oF —u 
a =3% 
oF =u 
S <u 
0 S| 
ff =a 
Cat 
tT -¥ 


XIMLVW NOISY3dS510 WILT YALNI 


at 
oct = 


D-JECETEE.O 4-3 S8002-0 


ae eater 0-38 2T202.0 > ede .0 
fil? es MWE.O =» - FO-FELSTO*.D 8=—«_ «SE Q-BEPOTOS.0 Oo 202, {0-31 1020820. i 
cz VSTESE.0 10-3Tf300%.0 00 3 ake £0-2£80T0S.0 a ereeers ' (a 
— a Y° eeers. — OD WSOCTS.0 =: FO- BT 1a0e 2.0 £0-3E£2TO9.0 BF ste 
OS BAMEOE ILO 10-3@£8205.0  f0-325TE6E.0 I0~3 28S886.0 3888 


———«FB=380G985.0 = 10-90085T 1.0 f0-38S8S25.0  I0-3080TSS.0 featsosice 


ae ~~ : i a is Ge 
ae ; ae A: oie 
- — fa Pa a ; : ee mee tay Y, : Pee a re < i a | } = all 
: 53 as ”, ; oe } J 
7 s Sica S= eae ee’ not - f a 
= = =A 5 : . | 
—f : i" , : j P oth é: 1o-3aa5tesa) 
m3 Y ; 7 ae , £0-JaSoS3s< 
: 9 a be r =a ¢ . {G JIQGNT 1.0 
. [> | .  sO+IS99%eT.2 
—. . . = -  . (= OG 28F9ear.0 
a | . | ns XESTAY WOTTAITAACS BATTS 
a : & -3 , =) a a a  *- = | ¢ <3 Ss «0 “£3 
ey  1O-38e8498.6 LO=FEES TACO Of SPELSE 4.6 Qo ab ho ded ' £0 Fa Teaeq 0 OO SESESS 160 OO sStePss.0 40 3990007.0 f 
> — «OG 3985 P2L 0 OO 324LiT isu O9 SSeiawg.d OO 3045. OO SPESTOS LO OG 3WPHSES.G IO 200000!.0 OO ISF8ear.o . = ‘ 
ee. Gf JelePsi. 9 Of -ISTESSI. 0 o S8S6c1k.0 o0 25 1)80..0 OO 3eTIos.0 IG 3200000L.0 80 BC00FS.O 00 FEr€asro & -H# j 
7 @O 21etitL.0 06 3€S08T1.0 00 3eo0e 45.0 OO Jeseese.o 16 300000: . 00 3TI#OS.0 O09) Seedees..6 60 3ffSeet.0 + --A 
-. 60 Deter s.o 09 JbC¢eTL.0 HO J22P205.0 10 2000001.0 OO 3#seess.0 , 20 ISTASOR.G GO WIGI£SS 20 OO S&0Gettsp 2 <2 : 
OO Ses€eer.c Of Fewveoer.o AG AIGGGSY.0 OD FeSPSss of O08 JeSOFSS.O GO 3858F15.0 ao 35 ELaAeS.0 OO ZFC SEL.O 4 : 
OO Wedget.a iG 320000! .0 / GO FCEOCL.2 Cf SECeSeT 1.0 GO FESOST £0 OF “34aseEses.o OO. 39e8sT 1.6 £0-3€58beP .5 , | 
i 200000 1.0 O08 26@sSel.d | GO FeaeHCl co OO S0eTers.o OO 3/4517 1.0 GO BELIE@OT. 09 3988071." [O-DSeseneg 5 Se | 
BO SOfeTe!.o GO 34eTTti.6 ' QO 3@#E8E] 6 60 3? 1Di¢).0 OO J0Se8TEL.0 OG 3JEESPII.0 00 JSfSESI.0 ie all e  =f8 
: e --9 . 
10~ 3asores 0 i, 
: OO JOT3EGE.2 § <8 = 
OO JE€SPS1.0 £ =H | 
; OO AsssTet.s 
f 00 FIstarea 
‘ OO SPRESEY.0— os 
: oo Estas i 
ag i 
=. + 
= ee : _—_" 
= : = 7 : » 2 = 
A 'g - = = 4 =— : a - _— 
aan ‘i ay 
- : ; ; 3 ; vo 
: : é ’ - Accel — hs 


235. 


T0-3629TLS° 0 
T0-321682 4° 0 


T0-3216824%°0 
00 398241 T°O 


T0-3S 12S82°0 - 


8 -3 


2£7918°0 


TO 300000T°0 
TO 300000T°0O 
TO 300000T°0 
TO 300000T°O 
TO 300000T°O 

0°0 
TO 300000T°O 

0°0 

0° 0 
10 300000T°O 
TO 300000T°O 
TO 300000I°O 
10 3000001°0 
TO 300000T°O 
10 3O0000T°O 

8 -2 


10-36 e41LS°0 

T0-34 T2568 2° 0- 

TO0-322S82%°0O 

10-34 1ZS82° 0- 

10-3S 12S82°0 
/i 33) 


6 


6 


10 300000T°0O 
TO 300000T°0 
10 300000T°0O 
TO 300000T°O 
TO 300000T°O 
TO 300000T°O 
TO 300000T°O 


0°0 


TO 3000001°0 


0°0 


TO 300000T°O 
TO 300000T°O 
TO 300000T°O 
10 300000T°0O 


0°0 
{h =3) 


00 300000T°0O 
10-372S608°0O 
10-34% 25608°0 
T0-3% 26608 °0 
TO-3€S608€°O 

I=) 


00 3€€EE6°O O00 300008°0 00 300008°0 


8 


8 


=(VAONV}1S3 313¥ G3ISVIGNN 8B8SBL°O =02 um 


10 300000T°O 
0°O 
TO 3000001°0 
TO 300000T°O 
TO 300000T°O 
TO 3000001°O 
TO 300000T°O 
TO 3000001°0 
0°0 
0°O 
TO 3000001 °O 
10 300000T1°0 
0°90 
TO 300000T°O 
TO 300000T°O 
oo = 3 


0° 
0) 
TO 300000T° 
0 
0) 


00 3£%12S2°0 

TO-3€4TLS8°0 

TO-3€71TLS8°0 

TO-3E41LS8°O 

10-32S82451°0- 
Sr 3) 


00 3TTITETO OO 38Z2ZT°O OO 3BLLLT°O OO JITTITI°O 
L 9 


0°O 
10 3000001°0 
TO 3000001°O 
10 300000T°O 
TO 3000001°O 
TO 3000001 °O 
TO 300000T°O 

0°0 


10 3000001 °0 
To 300000T°0O 


S  =2 


TO 38€029%°O 


TO-3€4%TLS8°0O 
00 3299992°0 
00 38€¢S6T°O 
00 38€2S6T°O 
10-3299999°0 
» =3 


00 3€€e€EL°O O00 300009°0 
L 9 


S 


S 


TO 300000T °O 
0°0O 
TO 300000T°0O 
TO 300000T°0 
0°0 
TO 300000T°0 
To 300000T°O 
0°0 
0°0 
TO 300000T°O 
TO 300000T°0O 
0°O 
0°0 
0°d 
10 300000T°O 
p=) 


=4i 00 37902¥%T°0 


10-3€¥1T2S8°O 
00 38¢€2S6T°O 
00 3299992°0 
00 38€¢Ss6T°O 
10-3299999°0 
€ =3 


00 3€eeES°O 
¥ 


77 


10 300000T°O 
0°0 
TO 300000T°O 
TO 300000T°O 
o°0 
10 300000T°0O 
TO 3000C0T°O 
0°0 
0°0 
TO 300000T°O 
TO 300000T°O 


O° 
0 
TO 300000T° 
0 


=3SW 00 39999T8°0O 


10-3€%1T21S8°0 
00 38€¢561°0 
OO 3sEcsst°o 
00 3299992°0 


-10-3299999°0 


(G3) 


00 3€eees°o 00 


€ 


€ 


ZEECES°O 


TO-32S324T°0— 

T0-3299999°0 

hO=32 9999950 

1T0-3299999°0 

00 3608EZT°O 
t 3=3 


00 3222229°) 


aNnmar wn 


XIMLVW NOISYSdSIG JTdWvS 
=NV3wW9 


00 JEEEET°O 
é T 


YILIZA SNVIJA JIdWYS 


TO-32Z2z7Z°0- 10-3688889°O— 10-368888°0— T0-368888°0- 00 36888%°0- 
2 T 


YOLIZA $193453 DIXIS ITdWvS 


TO 300000T°O 

0°0 
TO 300000T°D 
TO 302000T°0 


o° 
o° 
TO 303000T° 
OF 
0° 


TO JO0000T* 
TO 300000T° 
0° 
OF 
TO 300000T° 


TO 300000T° 
€VK=3 


0 
0 
0 
0 
) 
) 
) 
0 
4) 
0 
0) 


=8SW 


ere? oovopvvvevuD 


weeseesre 


wee 


ce 


00 316%€99°O0 =VSW 
TO 309900T°)O st -¥ 
TO 3000900T°9) 7Tt -¥ 
TO 3009090T°O €1 -¥ 
TO 300900T°9) et -u 
TO 3003909T°O Kitee=t 
TO 3003900T°O OT = 
TO 300000T°O 6 au 
TO 390900T°9 8 -¥ 

0°9 LD =o 
TO 309900T°9O oa 
TO 3J00000T-*) S$ <-u 
TO 300000T°9D + =u 
TO 300000T°9O € -d 
TO 300003T°9) 2 <u 
TO 300090T°O est 
5 = 
TO 3093900T°O St -3 
0°9 oT —-u 
0°0 €le=8 
0°0 et =u 
0°90 Tt -¥ 
0°0 Ole—3 
TO 300000T°0 Ce ts 
0°90 Sc 
0°9 Ea 
0°O Jao 
0°90 S$ <-¥ 
0°O FSR 
0°O | 
0°0 Coo 
0°0 YS 
te =3 
X1ULVW Viva 
COLSCESl LECT SSLSCESee 
e SNN& J IWdwvxX3 e 


BodDD 


ver 


et 


F 
'. 


ot 1 ; Secooes "0 # ocooere - sere to gooocer.0 C 
aie “ 16 3000604 .0 ; eens S ss Siac be ie = Chet 
< 40 3000001.0 10 300000f.0 “fa Seeseor *D 2.0 Oth. “eA. ee 
a a §0-9000001.0 . AS SAREE» ro 300000! > 10 ad00001.0 io 3000001.0 «= « {9 «3000001.0 ©=©=- FO: 300800! 
SS a pO BO0000 IO 6.0 9.0 6.0 ‘$0 3000061.0 §0 3000005.0 10 4000001. 
- : 0.0: £0 30G008!.0 010 oe °F a. ee = sens 
Aa re a; ~ Ded io 3c00001.0 6.0 tb ae ae, = Gee 
SS oes ss $B -Fe00001.50 =—s sD IHONHAs + @ 30 2000001.0 1 30e000!1.0 10 300000! .6 10 3000001.0 os 10 303 
: ae G0 = FH -9009091.9 «=F -9000004.0 10 3000001.0 IG 3000001-0 =O aaa: a ae, 
“= . 10 3e00C0r.0 = 10 30.00001.0 £9, 4000001.0 10 3090005.0° Sa a ” See 
=  FOBeCOF.0 10 3050001.0 9 3000001.0 10 3600001,0 jo s00p001.0 © fe 2000001-0 40-2030 
: “ 4 $0 3000003.0 1469 3005001.0 10 300000!.0 40 9000001.0 10 300000f.0)' 16 300Q000%.0 8 f 300 
$B. FO0000 4.0 $0 3000001.0 * @0 , {0 °300000f.0 “D.0\ 0.9 amek ice 
_ = * _ $8 3900001. 0 - 10 FO00901.0 10 3000001.0 | 0.0 10 300000L.0 ¥O 3000001-0 £0 3000001 .0 
7 - 4 — ™ i 
<—~ : ; =o 
3 fo 300950126 
ss : 5 ; IO 29c007!.9 
- £9 2039001.8 
. £0 9099001,° 
10 3600004. 
ES : 10 3639001.0 
. oe 
: £0 3999001.9 .8 A 
; : #0 3007001.0 «© <A 
| i $9 -2003003.0 -01°-9 
5 é #0 3055004.0 IL <A 
F ) - io. S00900%.2 Si ~“@ --— 
= 4 2 WQS005450 FI 
: 7 0 A0D9GD Ex? ares 
; os cates ee 
TeatSeD =PAVOKADTZ9 JAR GIPAISMU BSBST.G =OS NH 16 ZELOTS#.0 <F 00 ict the: 0 =32h 00 3008018-0 <a2M. 90 
2 e. . ¥ a : age 
oo SEE IIE-0: OO SBTTTL.0 COLRSTTTI.0 OO 3£15241.0 10-355555.0- 10-308888.0~ fo=398888 
~ : re iam 2 = 
_ ; a : a > : ‘S or 4 a i: ae a . 
i ae 00 BEE EE. 0 60 300000 .0 00 300008.9 009 SEECET.O- oo +300003-0 0° "seegERD 90 
= : + 7 ar a - a _ me 
: : ee Fe ey 
= 8 -~3- as : =3 6 =2 e J : =, ee a —— ave os ie oa: < re 
% fO-321T285.0° 10-3017 88S.0 [0-38 eROkE-6 fo-at28sal.o- Jo-ava3sa3.0  1O-3yee8e0.0 10=3783800. 
uv OO adaseti.o Heated fO-3aSzeok 0 JO-3ERINeRLO- 00 SBESER ELD OO F8ESPOL.O 00 3Tdsaes. 0. “fo-at aad 
POWSST2RS P.O. O-EF ens 4.0 fO-Fa eR 08.0 fOrs£Ai123.0 OO 58ES8CF.0 8 ©=— 0 SY aadas.0 90 3geS@?.0  10-358868e-3 
10-3ST2a0 9.0 10-32 1F285,0- 1Q=3eS2008.0 [O-3ESIVSS.O 06 32Teeess.0 00 38Eseei.oe 00 38ES301.0  (0-303ae08.9 
iG-JESeite. ~F2SH IV FeO 7 BOGC0COL.0 - 00 3tefies.0 PO-FEALTAS.O TO-ZEASIT CS. 0 JO—3E ST Tea lO-3VC3SeF.9— << oo « 


236. 


10-3624T2S°0 
00 36241L1°O 
10-3S12S82°Q 
10-38S82%71°0O 


10-32 S82%71° O- 
10-35 12582°0 
00 3624ILT°O 
T0-38S82471°0O 


98S82°0 


10-30T8€E2¢S°0 
10-38 S82%T°O 
10-38S824T°O 
00 3%2S6602°0 


=1¥S/WOH 


STOTO°O =31VWILSS 


10-3 22S824°0 
10-3624%1LS°0 
T0-36241LS°0 
00 300000T°O 


89682°0 


=j4309 ALI3SN390W0H 


TO-3€S608€°0O 
T0-32L5824°0 
10-34 TLS82°0- 
10-342S608°0 


T0-3€S608€E°O 
10-32L582%°0 
10-3225824°0 
10-3425608°0 


1989€°0O 


=3430) NOILVUNIVS 


TO-3€S608E°O 
00 39824711°0O 
TO-3¥1TLS82°O- 
10-34725608°0 


11SZ0°O =VAONY YIGNN SILVWILS3 13Y GISVIINN JD JINVIYVA 


ZBETO°O =3LVWIIS3 814%E0°O =VAONY Y3ONN JLVWILS3 VidtV 3D JINVIYVA 


s NOISU3SdSIO I TdWvS 


10-3299999°0 
T0-36241LS°9 
TO-3LSBC¥TPO= 
TO-J0TBEZS°O 
T9-37LSB7%7°O 
TO-3€5508e°) 
TO-3ES5508€°9) 
10-3€S698€°9 
20-31 8¢6296°0 
Sa 


Z0-3L8E256°9 
TO-3S1TLS82°9O 
T0-3S12S82°0 
T3-JES608E°O 


aANanrnoraoer 


om On 


fk ha 


Ss ed ees PS an pecteomery i 
- a é _ =a _ : ¥ ay ; a c i as ' 
ae = a | Se ae = 1050.0 saTAKtTZa see 
> : - hie 


- + a > oe “gemeyso =TARVROH ‘gaees.o | =39303 ¥¥iamaer 
: - 7 o oe ee ae oe i 


ey 


237. 


10-325S9ST°O 
00 380L29T°O 
T0-3SZ01L2°0 . 
10-3995S56€°0 
10-352240€°0 
10-3S51S9%7E°0 
10-3282282¢°0 
10-34718092°0 
20-35 %6166°0 
$$ =3 


10=-316294%T°O 
T0-3S20222°0 
00 3662022°0 
10-30 0£6247°0 
10-3%%f£28E°0 
TO0-30%SIT¥°O 
10-38602%¢°O 
10-3£6SS0€°0 
T0-34%7822T°O 
ho a 


20-3€6666°0 
6 


00 361088°0 00 3L% 
6 


TO-3€L€92 °0 
8 


TO-31 9L9€7%°O 
10-399SS6€ °O 
10-300€624°0 
00 388t%7c°O 
10-34% 1626S°0 
10-3026629°0 
10-32 6002S°0 
10-378 466%°0O 
10-3669681°0 
Sa 


8 


10-301 4%0€°0 


L 


G6L°O 00 36€229°0 


L 


T0-32LZ20TZ°O 
TO-3SLL40E°O 
10-3 %%E78E°O 
10-34%1626S°0 
00 36S18%¢°O 
TO-39LEEES°O 
TO-3L9EL64°O 
T0-340816%7°O 
T0-30S268T°O 

Ss -3 


T0-39EL€9 °O 
9 


9 


10-398€S22°0 
10-35159%E°0 
10-304S11%°0 
10-3016629°0 
10-39LEEES*O 
00 3aeeESr2°0 
10-35 1808%°0 
10-3%6029%°0 
10-3€ £686T°0 

a 


S 


00 32062£S°0 00 3%S%9%°0 


S 


10-386S5S°0 


TO-3LELEBT°O 
T0-3L872282°0 
TO-3860L%E°O 
10-32 6002S°0 
1O0-3L9EE64°O 
10-35 1808%°0 
00 3600€2¢2°0 
TO0-32L28249°0 
TO-32 18SlTt°O 
aaa 


T0-3589%9S°0 
Vy 


00 32STe7°0 
YW 


YO-318ETty°O 


TO-3SLYC9T°O 
TO0-37T809C°9 
10-3€6SS0€°0O 
T0-378%7659°0 
10-3408 167°0 
10-3769L9%°O 
TO-32L7824°9 
00 31686ST°O 
TO-3SELE0Z2°0 

(Ac a) 


00 359%S0T°O 
TO-3L5S9S1°O 
T0-315294T°9D 
TO-3T9L9EZS°)D 
1o-3zL20T72°9 
19-396cS22°) 
TO-3LELEBT°O 
TO-3SLYE9T°O 
ZO-3971TeLl°O 
6 -) 


ZO-39CITTELPO 
Z0-35476166°9 
10-39¢82e1°9 
10-3659581°9 
10-3052 68T°) 
TO-3et6B61°O 
TO-3L13S21°O 
TO-3SELE02°9 
10-3651898°) 

<8) 


ANAT noraer 
J 
i+ 4 


aANMENOE OD 
| 
« 


XIULVW NOISY3dSIG 1931 NIHLIM 


€ 


Zz 


19-328%8%°0  20-3€8028°0 


1 


SZINVIBVAID WIL] W311IW8id 


00 399SEE°O 00 30856T°D 


€ 


Sstost 


B8E£949°O 
1€96S69°0 
TO 33cLSE1°O 
TO 36€0€92°0 
10 AL9LB6E°O 
TO 3890S4%%°0 


VEREELVCLOLELCELLSLCLLLS CEE LEC SEL EL OLe 


© GOHL3SW Wiivuvd AS S 


eee 


eecree® COEECEEL ELL PCLECELECLLSLLELL LEY 


wees ee eer ve 


2 


10-3L£096°0 


T 


YOLIZA NV3IW 


s3asv3 30 ON 


Ocun 


ALI WEVIIWSY 
ZONVIUVA vOuNs 
JONVIYVA 3NUL 


SINVIUVA 
NV3W 


ODVPLOLELOECLE LCE CELLS 


YZLBWVUVd NIILVINdOd JO NOILAVHILSS e 


DEDPDODLOPLBPLLPELLELE 


| $oe98aete0.0 


“JO-IVSssa5.0 


LO~32T 2492.0 
10-3eTT #0820 


1G-380880E,0— 


16-32 4057S. 0 
OO 38OTSéi.0 
FO=3T f2e0!,0 


0 


--<3 
16-36c8SSf.0 
{0-9zE0220F.0 

1-28 POT AL, i) 
£O-30 0% 113.6 
10-30 sES8E.9 
10-Jooeesa.o 
OO. 322°5685.0 


2O-32S0TTS.0. 


(0-91 OT S01.0 


10+3e¥ cad 


90 wilens v0 sete 00 38e5t0.0. pe 


Te 


2 =3 
10-30 0Q081.0 
10-39 62° 02.0 
§ro-3F 00Se.0 
{O-40Tresa.a 
10-30 7S e2.0 
OO 2R8£P4e.6 
LO-300fe $2.0 
tO-~daaezer .5 
IG-3LSTSES .0 


f 


—— wi 
eiinetiriae 10-308 


2 +3 


- $0-308S54.6 


10-2808 10.0 
1O0“3T d£4 eb a6 
£O-FSFELES 60 


GG 38e2tees.0: 


sO-Zeleged.6 
LO-3 del TSE 29 
i6~3 eT (agt.0 
Ei- TLV SOLS .6 


rolren hc cae 


Bod 


; 3 : ~~ ao Cees 
= sim: As | = > 
27% Sais + Sora sll 
=~, 7 gel .s ; 
i - oe 
oe 7 mal aa 7 
= 7 aa = 
- 7 es! 
re a ‘ 


ae 
4 ed ies 
Feo. soamtet : 

* 3 . & <3 
10-28 CHOI .0 I0-3f 16275 ,0 
20-30 PNT d4.0 ‘£O~9STSESA.6 
£6-32 18068050 QO 3POBESS.0 
OG. SEFERAS 0 fO~30 [6988.0 
sO-36fE 602.0 16-31 OEECA.0 
£0+207 €e$a.0 10-38 C60S2.0 
£6~2Gadi10.0 10~FeeOvet.o 
20-3232 SKE .0 i0~37 85565.0 
i0-398f 254,09 10-3TETERL.9 

- 
a 
_ ec be 


= 


eeeetreies 
OO Sheaeer.o 
PO“7STSESOLD 
ee 
f9-3e08 1 A,0 
LO-BasEC 7 O.5 
FO~JECZ2A0E LO 
[0-39 £8085.) 
10-32Tard1.0 


iG-3P2028t.9 
{0-39S8551,9. 
SO-3840P 102.5 


SO~3eSs 127.9 


238. 


1Z00W 


Z0-30294%S9°0 
Z0-3L6L8€8°0 
T0-378L60T°O 
TO-307E80T°O 
T0-38262T1°O 
10-3699TTT°O 
TO-31%290T °O 
Z0-3208LEL°O 
20-39T£06S°0 
G3 193d X3 


Z0-3061259°0 
Z0-3186188°0 
10-324Sc01°0 
10-366%20T°O 
T0-3S24%L0T°O 
To-3292TIT°O 
10-3980T0T°O 
20-3201269°0 
ZO-ATETSSS°O 
SINVIUVA 


€O-3TEL9INY°O €0-3966T92 °0 

10-39S68€8°O 10-38 %129°0 

10-30S708Z°0 T10-3€09€61°O 

031933dX3 03A¥3S80 
SINVIUVA 


| 
1 
{ 
\ 
\ 
\ 
1 
J 
\ 
| 


*4°W YBONN S3NIWA 03193dxX3 ONV S31VWILSS 493333 93X! 


1 
| 
{ 
J 
{ 


00 3999S8E°O 
00 31S609E°D 
00 37l8LzLT°O 
10-3050S08°0O 
10-3548562°0- 
10-391S629°0- 
00 39583ST°0- 
00 3022%62°0- 
00 3€3786E°0- 
03193dx3 


00 36089ST°O 

TO 310881T°O 

00 3Sl0€%%°0 
03193dx3 


09 3959S8€°0 
039 3840T0E°O 
00 3298LLT°O 
19-3189%708 °O 
19-343 8662 °O- 
19-3£02629°0- 
00 36168ST°0- 
09 3£06%62 °0—- 
00 30SE85E°0- 
NV3W 


00 37502ST°O 
To 20988IT°O 
00 32S1S%%7°0 

G3A¥3S80 


NV3W 


ANATNOR OD 


iuvd 


4 dOS SOIISTIVIS JATLdI ¥dSIO 


youds 
SW31I 
433fF8NS 


7300RK VAONY Y3CNN S3NIVA G3193dxX3 CONV SAUVNDS NV3W wOd SIIASTIVAS FAL Ld] ¥9S30 


00 3€€€802°O =TWAYSINI SSV19 


0°O =WNWNNIW 


zo 390003T°O 


eeeveseescere 


=WNWIXVW 


eeeecee 


© indinO 3O AYVKWNS C 


erpeepvorrne povvvr2 


veers 


veeeesre 


VIC CCEE 


259% 


ZO 3000092°0 ZO JELITIESO ly <u 
ZO 309000S°0 ZO 396382S°0 O04 -¥ 
ZO 300009S°0 zo 3988299°0 6k —u 
20 30000728°9 ZO 3698S%L°) BE -¥ 
ZO 30000SL°0 ZO 3S9SLLL°)O LE <¥ 
ZO 30009028°0 ZO 3582L9L°9) 9€ -u 
ZO 30000TL°9O ZO 310%22L°9 GE -u 
ZO 30090$L°9 ZO 30€3699°) ye <u 
ZO 3000062°0 20 3865299°9 ce -u 
Z0 300002%°0 ZO 32S574%€S°0 ze - 
Z0 399900S°9 ZO 342L29%°) le -¥ 
ZO 300009€°0 Z0 3£83S50%°) Of -¥ 
ZO 30000LE°0 ZO 30956%E°0 6z -u 
ZO 300906€°9 ZO 329c00€°) 82 -d 
20 3000052°0 zo 30092S2°9 12 <3 
zo 3000092°0 ZO 3€S%612°9 92 -¥ 
. ZO 309009T°0 ZO 3EBTLet°d G2 -& 
20 30000ST°0 20 3€6S5ST°)D 2 -U 
20 303009T°0 ZO 3S809€T°)D €z -u 
zo 399008T°O 20 3TOT9TI°O 22 -u 
10 3000008°0 TO 34%E166°9 lz -u 
TO 3090006 °2) TO 32ISsl48°0 oz -u 
TO 300)00S°0 10 3205S2L°0 ot -¥ 
10 3000008°0 TO 3985129°9 eT -¥ 
TO 390900€°O TO 3EZT%ES°9 LI -¥ 
10 300000%°9 10 36€%6S%°9 91 -u 
10 300000€°0 10 3006S6E°) STI -u 
TO 300000T°0 IO 3LSLTyE°)d ol -u 
TO 300000S°0 10 36099652°9 €t -u 
10 300000%°0 10 3621992°0 zt -u 
10 390000€°O 19 3S8€7¢22°0 Il -u 
0°0 TO 3LbveoT°) Ot -u 
0°0 TO 389S891°O 6 -8 
10 300900T°O TO 39LTL9T°) 8 -u 
0°0 1o 3672L821°9 1s 
0°09 10 38982T1°O 9 <4 
0-9 00 3212066°) S$ -4u 
0°0 00 379L0L8°) > -' 
10 3000002°0 00 3ZTTL9L°9O € -¥ 
0°0 00 30€5949°) z2o-w 
0°0 00 31L¢€386S°) to 
2 -3 1 -3 
ALIVLAVI AY 40 NISTYVdWID 
1O-329T6%E°O TO-3TLE6TIE°O { 00 3€06209°0 00 3S2666S°0 30d 13u 
Q3193dx3 Q3A¥N3S 80 | 03193dx3 G3A¥3580 
SONVIUVA l NV3W 


S3JLVWIASS ALITIGVITZY YOI SITASTAVIS AAT Ld 1¥IS3G 


10-30000S2°0 = =IVAYNSINI SSV19 00 3090002°0- =WAWNNIN 


00 3565656°0 


=WNWIXVW 


IO 3LOIZZS°O =2°O0- MOTZG SILVWILSS ALIVISVIIIY 3O ADNSND3Y¥S G83193dK3 


(G3SVIG*VAONY)VINWYOI VHd TW NO G3SVve SI NILAVWILS3 


SEALER LAE AALS BALAL RTA LABS 
e AGNLS ALIVIGVII3Z8 e& 
CECEDEPEPEELELLELLLLELEELS 


Fo-s000025.0, — ANT 2K 00 303 
| ' ps ~  @ 


3 YTISE ca BT 
r re jig 
- @37929x3 a ave 

0 “358100810 “so-siteeie. 


ape TS 


10 390000S-5 OG -BRILTATSO 
eh: «0G Searere.3 
ft. OY aSrsoce.> 

O.0 - (8 FH3bSTI.9 

O60 °° 10-3RS58EI.2 1 
$0 -3007001.0 10 3etéTes.> 


~<a | ae a 0.6 10 386¢803.9 
con eet) £0 3TPPECELD . OF eR 
i -e : #0 S)00086.0 19 FEBESSS.0 Lf 8 
i @:: iO FCG0Ob).0 (0 FTLa7S.9 8 SP -F a 
= . : Re ee ee a 
o% 4 £0 AvG00D4 .0 (0 GEPTPRE.D OF <M 
_ . y 10 J00008€.0 40° 7uGeeee.3 Si = 
£O Fenooos.5 —e JeEaP2e.5 8 8=60at = a 


; 40 AQOIDGE.0 . ID -BESPRERLO. T1 <0 Er 
. £0 FO09008.6 £0 3asetse.0 oo 

| $0 2009000.0 £0, 250a2¢¢ <0 a - 

: . 10 39an0be.9 10 aS f2bee.0 ~ 
10 3209008,0 £0 aseeseo.o 


age - SO F939081.0 so 3i07eT1.9 
a “$0 20700e2-0 SO Seaneer.o 
a ae SO 3000081. 0 
ee $0. 3039081 «0 
ee : $0” 30900a5.6. 
a —- ; SO 2000025.) 
a ae ; = oe , SO 39. ares. 
ra ae : so 3u 


— a E 
ot) == aie = 

* = = — S : 7 = 
—— : 7 is @ = - 
a ~ 7 toe a a 
oa 7 = - = - _ = 
tat * _ en = ; 
a ; | 5 £0: 3000058 
“= — $0 Sontag: he rs 1 

; : s . | SO 305000e.0 . “Sa 3925882.0. 0 


7 _ | } ;  -§ Sopebat.o —~ SG Srettaeco 


240. 


99996L°O ={1S83)98 Y3addN 
276808°0 =ONNOS Y3ddN 


228062°O =(153)8 YIMOT 206°C 
1S6S262°0 =QNNOS ¥Y3M01 


62 


=@ Y3ddN NVHL YAI1LV3ND 


"ett 


=340 


°oT 


0°O 


TO 300000 


20 300D0TTI°O 


O°) 
40-39%7996S°9 
Z0-3669961°0 
00 3ESLEET°) 
To 399€09T°3 
TO 320S€%7L°9 
20 39S9S51°) 


£ 200°% O% =8 Y3SMOT NVHL SS31 S3SV9 30 ON 


=Vd4I 0S0°O=(HIV3)T3A3RT9IS S3JIVWILS3 VHd IV 


-u 
=e 
-u 
-u 
—u 
=o 
-u 


? 


Pe _— 


-_ 


ee | 7 > -_ es re i eo + > : = = ‘oe: 
‘sea0s.0 egvuoa A399 leendia =onude AaWOs | “Sts. 0 - 0.0m: 
:  apeaet eg w(T2308 AINA —TSEORS.- mavaaee RAWOS oe.) oes 8 maaeu WANT 937 sin weak 0 


- ot — 4 a a a 7) epasd? 

es * * ; - 7 
1 eats = a i ” : 7 - — . - — _ 
—, a2 ' : : 2 : 7 ee) 

7 2 ; Z —h& 4 —- 
7 a. Ye’ ' F d Ma : - 
7 4 a> Ox <  —— 
< 


7 co fees ol x i 
< — Y ee al i ‘ { < 
po 2 : 
_ i - t oi 
_— Ld 
~— ~~ i f X nS \ 
ae 7 
—_ y A = ~ 
== - = 
= \ a? 
—_ 4 
a a 
a 
. 
- 
= 
" r - i 
/ 
= ld 7 — 
— 
— _ 
_ = 
— 
as 7 ~ - _ 
8 ‘i = 


s _ - ry 
» 
$ — => a 
= ~— — 
— ok - _ 
a = 
~ ve. i "4 — —o a 
j ied a 7 : -_ 
aoe = a _ 


vr 
t 
} 
€ 
GS. 


241. 


O°T 6°0 


Cad Sal ed eld Ol Odd dd Od dd od ed Odd dd 


* 


8°0 


L£°0 


9°0 


6£%70°0O 


AINIOd ONIddVTY3AO (#) 2034s G3AYN3SED (e) $03¥s O3193dK3 (°) s31L0N 


S°0 


=80ud 


Te 


7°0 £°0 a0 T°0 0° 0= ieO= 2° 0- 
ed Hed Ma et a at 9 Zl ee 134 
* ovete s & 4 2™s * { 
*°+ 4** * * 
"°° * 


1-000°01T 
(3 DAININDAY4 
=J0N 20 3LL19S%°O = ==IHD s1S31 LI4d 30 SS3NG099 OS-IHD 


eee ee Tee EE ESTE ERTL ETE STEEL ELE SEIZES 
% NOILNGIULSIGO AININD3YS 4D 101d & 
(TST TST TITS SISAL ITI TIS TTT I TPT PTs tT5 


. fs 


» 
f= 2st ist-1-f 
Oo. 


i 


e.0 


* 


ed Sg -{-= =f={~-1-1- terete 
8.0. 


Ted 


4.0 


‘ * 2 s ¢ * 

“Sea lele 

1-{-t-I-{-1-I-1- =|-t={+i-[= =t- <i {>{= we te =<(-i-i- 
720 $20 £.0 Se ) 


. THEOS DRISIAIASVG (+) 70384 oavaseso (4) 


* a 


={= ae Sie Da a 


O50— = Le0- s o- 


70253 a3t939K3 Gad 


242. 


ZO 300006S°0 ZO 3€427099°) ly -¥ 
ZO 300002L°0 ZO 399€0Z78°D 3=0% <4 
ZO 30000S6°0 ZO 38%9258°3 6€ —¥ 
Z0 3000066°9 ZO 3004106°) SE -Y 
Z0 30000%8°0 ZO 309TESB8°O LE <3 
20 30000%5°) ZO 36LL9LL°D) = DE ed 
20 3000062°0 20 ST8EEB89°O GE -¥ 
20 30009LS°0 20 360506S°) oe -u 
ZO 300009%°0 ZO 3106295°0 €€ -u 
20 300008€°0 ZO 30€0%24°0 3 8=2€ —¥ 
ZO 300008%°0 ZO 3TL9SSE°0 Te -¥ 
ZO 300009€°0 ZO 31£9962°9 O€ -¥ 
Z0 3000062°0 ZO 36S19%2°0 62 -¥ 
ZO 3000012°9 ZO 3€2%402°0 82 -x¥ 
z0 3000002°9 ZO 38TL69T°) 12 -u 
ZO 30000ST°0 ZO 3€86041°0 92 -¥ 
ZO 30000ET°9 ZO SEEZLIT°O SZ -¥ 
ZO 3000001°0 TO 322€925°) bZ ud 
TO 30000C9°9 TO 3259%18°9 €2 -u 
i TO 3000006°9 TO 34541 189°9 ZZ <u 
TO 300000€°9 TO 386L04S°9 TZ -u 
10 300000%°0 10 3T0S6L4°0 OZ -¥ 
TO 3000001°0 TO 398LE0%°O0 61 <4 
TO 3000009°9 10 3S8304E°O BI <u 
TO 300000%°0 TO 322588Z°D) = AT sor 
TO 3000002°0 10 38€3%%2°9 9T -x 
0°0 TO 32872802°0 St -¥ 
TO 300000T° TO 38S92LT°O oI -¥ 
0°0 10 32051ST°9D €t -u 
0°0 TO 32TZ0€T°9 zt <u 
0°90 TO 3¥8B8lIT°O It s-¥ 
TO 300000T°0 00 3389€96°9 Ol -¥ 
10 3000V01°0 00 31z0zE8°0 6 -4u 
0°0 00 3S9561L°9 8 -u 
0°0 09 38L%%29°) ao 
0°0 00 351324S°9 9 <3 
0°0 00 34732L%°) g¢ <4 
0°0 09 3€IL214°) > <4" 
0°0 00 3S89T9€°0 € <u 
10 300000T°0 00 31¥”9TE°D 2 -¥ 
0°0 TO 309SE9Z°0- TT —¥ 
2 -9 t -3 
ALIVIGVIT3Y 3D NISTYVdWId 
TO-3L259SZ°O 10-3552S€2°O | OO 3TE96S9°0 00 35012S59°0 499 134 
03193d x3 03 Au3S80 | 03193dx3 Q3Au3S30 
SINVIMVA | A NV3W 


SSLVWILS3S ALITVIGVIIVSY YOd SIIASITAVLIS 3ALid14u9S30 
T0-30000S2°0 9 =1VA¥ZINI SSV¥19 00 3000002°0— =WNWNNIA 00 3566655°0 9 =nNWIXxVW 


TO 3LOT22S°O §==2°O0- MOT3D SILVWILS3S ALIDIGVIT3SY JO AININDZYS 03193dxK3 


(GISVIGNN*S VAONVINITLIZYYOD JOLSTYyN NI G3ISVG SI NIIIVWILS3 


COC Coe Bee Vee meer Ce ee 
. e AGNLS ALIVavVIIIY eC 
CULO LUE eee Cee CO CTe 


i ; ar * 
~ asvamras rn sieal a 


Pal a many — me = 


(Tse i ees. 
| ? «HOM BTSEVAS.0  10~3RESE +e 00 2180P23.0 aa 
; / jim : 7 — 
at a ; ie : (eo: — ae cs ha _veraten 


cent 
Bg 


A ae 040 

io 3900001) 

Be a 3, 

¥0 3000005. 

\ 4 10 sash 
50 3000608.9 


19 32930a£.0 - BL <8 


—; {0° 3609001.0 - 40 3enve00e.0° oF -a- 
_ 10 300000a.0°° 10 Fi0zeys,.0 0S A dla 
E 5 . . 10 300000€,9° fo yaefote.9 45 A _ 
Fs - . . {D0 3000D5e.9 19 Baeried.d2 $s =A 
: 10: 30000C2,9. £0 3823018,.9 €S =9 
7 ; $0 2000001.0 Pb astear?. > S$ =A 
oe - “$0 FOQGSET.2 SEESTITD : BS A 
oa: = SO 30000¢!.9 Sp! 36808) .0 as -8 - 
i ; $0 2009005.9 $0 -281fPer.> TS =A a 
a SO S50000%5.9 SO 3€80005,9 8S -A ne 
—. i, SO 7009025.0D 50 se@iaes.0 86S = 
—— : . SO RD00D9ELD SO 3¥£9085.9 0€ =A 
=e ; : SO 390008).0 sO 319 3e8€,0 ft : 
($0 209000BE.0 $0 3DeGeSS.O 
SO 3000060.0 $0 4£00S9E.0 - 
. ; SO 30000T?.0- ae SECS E69 
| i £6 s0900eT oD 
“ £0 3O09oRe.9 hart.) 
be : So 300008860 ged e 
: Sg 2000028.9~ 


: 50. g0000erLD 
SO 300008F.0 - 


ae ay Ee 


243. 


#1LS28 °0 =(1$3)9 U3ddN LETZOE°O =(1S3)8 YIMOT 206°C 62 =8 YaddN NYHL Y3ILVSYD * 200°% O¥ =G ¥Y3MO1 NVHL $S37 S3S¥2 30 ON 


9€29€8°O =ONNOO8 w3ddN S19E9E°O =ONNOB ¥Y3MO1 "zit =340 a =v 40 0S0°O=(HIV3)T3A37 DIS S3LYWILS3S VHdl¥ O3LSNFOV 
0°0 470-397096S°0 84 -Y 

0°0 0°93 (EP | 

0°0 T0-3647L812°9 94 -U 

0°0 00 386€0¢8°) SY -¥ 

TO 3090002°9 TO 3L%8T¥9°)D 94 —u 

2 ZO 3000021°0 22 396L4%T2°) Cy=3 


ZO 390008E°0 ZO 3e2T8e7°O CAA: 


Pe a a xy 
ah | 
= ee ae 
a Y ae mF 
Gh Se 
b ’ key 
4 | 
6 


onvd= 6 


§ BLE2LD= OF 
bEB FOO 


bee 


= 


ft 
re 


2930. 
¢ 


wie 


f 
i 


1 
“3ast 


9 

4 am 

j= 
7. 


E2h 


= 


7 es BONMO= O° 


© 


“S*aog romex af 


i, 


a 7. 


‘a 


- 
_ 


it Mb 
Td Ok 


te 


20 


.t 


a 
a°a 


: a ia 2" (tas 


244, 


o°T 6°0 


I= IP b= bad $1 1-1 b-bd =f f-b 1-1-4 -f- 1-1-1 f-1- 1-1 -t-t-1-t-1 


cd 


8°0 


L°0 


9°0 


o # 


809€°0O 


ANIOd ONIddVVY3A0 (+) £03543 G3IAYZSED (e#) $3344 031936KR ¢°) 310N 


s°0 


=80ud 


82 


7°0 €°0 C0 T°0 0°0- T°0- 2° 0- 
-I-l-I-1l-1-1-l-l-1-1l-t-I-1 734 
* See ee Se ig * 
*+° ° & 
ae a * 


-000°T 


-000°2 


-000°€ 


-000°4 


-000°S 


-000°9 


“000° 


-000°8 


-000°6 


= = oe 
oe we ee es ee ee ee eee OO a OO Se Se 
—_— 


! 
1-000°01 
(3 2AININSIYS 
=JQN 20 320S00€°O0 =IHD 21$31 114 30 SS3NG939 OS-1HD 


SSSSESERELEELETETSEESETSSELTTSZLATE 
% NOIANGIYLSIG AININOZ84 4d LOId 3 
SSESEETEEETAREESRSSELATSALALTSETTE 


oe 


= 
4, 


ES 


© 


te 


EciEd 


ey Ex 


} OAEST¥SbiNe bO IML 


t ta) 


ve 
em 


 & 
‘ 
oe 


ara 


ote 


SS 
. 
~ 


CR2EBAED EMBO t+ 


o- 


So 


F°G 


O73 


o*s 


one 


> 


=fot=1 1-9-1 = 1-8 1-8) - |= !-1-1=1=1-1-1-1-1- l= I=) 1 bib 


= 
my 


Al 
<4 

iis 
Ss 
pry A) 
= ‘a 


\" 
gin 


ne nee 


or We 


i 


om 


ne : 


es 
‘a 

‘a 
cer | 


* 


até 


he 


ee 


re ah 


ui 


i an 


